1446341 - The XUL "tree" widget will be removed from mozilla-central

Status: NEW

(Thunderbird :: Upstream Synchronization, enhancement)

Description

[:Paolo Amadini]

As noted in bug 1446335, as part of the architectural efforts to remove XUL from mozilla-central, we are looking into deleting the "tree" widget entirely.

While this removal will not happen very soon, the eventual plan of record does not involve developing a replacement that provides the same user experience. Instead, each of the current uses of "tree" in mozilla-central will be evaluated independently and replaced with an alternative that works for the specific use case.

Keeping this in mind, I expect that Thunderbird and SeaMonkey will also have to develop alternatives for the portions of their user interface that depend on "tree", or adopt the technology into comm-central.

Comment 1

[Jorg K (CEST = GMT+2)]

Thanks for the heads-up. I believe we've already looked into a replacement for the tree widget, but I'd have to find where that was done.

Comment 2

[Jorg K (CEST = GMT+2)]

Found some stuff:
- Bug 1183812 comment #3
- https://mail.mozilla.org/pipermail/tb-planning/2017-March/005322.html

Comment 3

[Ben Bucksch (:BenB)]

http://benbucksch.github.io/trex/fastlist-test.html

Comment 4

[Ben Bucksch (:BenB)]

XUL <tree> is completely central to Thunderbird's UI. The XUL <tree> was originally implemented specifically for Thunderbird, and with the needs of Thunderbird in mind. The folder pane and the thread pane are XUL <tree>s, and the thread pane is the most prominent UI of Thunderbird.

It is highly complex and needs:
* Fast enough for up to 1 million entries
* tree: hierarchical entries with expander
* table: multiple columns, sortable, configurable
* the combination of tree and table at the same time
Without XUL <tree>, and without a suitable replacement that at the very minimum does all of that, Thunderbird will simply be dead.

@Everybody: To start, it would be good to compile a list of requirements that the new implementation needs. The above list is just the big picture, but there are many small requirements to be worked out.

This will not be trivial, and won't be done in a weekend. I've already looked, and there's nothing out there that even comes close. This is a 6-12 months project with full time staff, given that so much of Thunderbird hangs off it. We need a suitable replacement that meets all requirements of Thunderbird, *before* XUL <tree> is removed. Otherwise the project is dead.

@Paolo: I understand that this is just a placeholder and you don't have a timeline yet. But this needs a concerted effort to create a replacement, and is a large project on its own. If you want the Thunderbird project to handle this, we'd need 2 years of ahead warning time, and funds to create the replacement, and need to build a development team for it etc..

Comment 5

[:Paolo Amadini]

(In reply to Ben Bucksch (:BenB) from comment #4)
> @Everybody: To start, it would be good to compile a list of requirements
> that the new implementation needs.

I like the approach of taking a bird's eye view at the requirements of Thunderbird as an application, that is followed here and in the linked Thunderbird newsgroup post, and I would go even further in suggesting that a modern user interface for listing messages and threads could have different levels of similarities to the current list with the tiny expander icons and a single overarching scrollbar.

In the end, what this will look like in two to three years is a decision for the Thunderbird community to make, both from a code and interface design perspective.

> @Paolo: I understand that this is just a placeholder and you don't have a
> timeline yet. But this needs a concerted effort to create a replacement, and
> is a large project on its own. If you want the Thunderbird project to handle
> this, we'd need 2 years of ahead warning time, and funds to create the
> replacement, and need to build a development team for it etc..

It looks like the removal of the tree widget has worked as a good conversation starter, probably because it is demonstrably achievable and it does require some dedicated work to replace. However, as you noted in the newsgroup, this is just part of a bigger effort to remove XUL from mozilla-central. Just as an example, bug 1444468 is related to the removal of XUL overlay technology from the platform, which will happen much sooner. This recent thread lists other parts of XUL for which there is an action plan:

https://mail.mozilla.org/pipermail/firefox-dev/2018-March/006240.html

All the little fixes needed as a consequence will likely require more effort overall than a new interface for listing messages. The same long-term planning considerations for Thunderbird also apply to these changes, and I can only anticipate them becoming more pressing as the XUL removal project gains momentum.

Speaking of ahead warning time, my general consideration is that in all communications we've been quite explicit about the intent to remove XUL for quite some time now, and it doesn't certainly come as a surprise, although only more recently this idea started to be an effort concretely led by engineering. In fact, this is the best time to make a Thunderbird counterpart to these concrete engineering plans, and take decisions.

As an engineering choice, we already took the explicit direction of removing the XUL code from mozilla-central entirely, doing this incrementally as we remove features. Given this, I believe that adopting portions of the mozilla-central platform code in comm-central may be the solution to gain the lead time you mention here, and may work for the "tree" widget. This may take different forms, one of which is branching some or all of the top-level folders and maintaining vendor patches. Of course this is just a suggestion from the perspective of someone not deeply involved with comm-central development, and there may be different options available, which Thunderbird, SeaMonkey, and other projects based on mozilla-central can evaluate.

Comment 6

[Ben Bucksch (:BenB)]

> Speaking of ahead warning time, my general consideration is that in all communications we've been
> quite explicit about the intent to remove XUL for quite some time now

I have been constantly - both outside the TB Council and from within - telling everybody that XUL will soon be gone. Every time, a discussion unfolds about how Mozilla has announced this since a decade, and that it's still years away. But despite all my efforts, nothing moves.

The lack of a clear, definite and reliable timeline is the very thing that has blocked Thunderbird's progress. We need to know: XUL overlay will be removed on March 2019, XUL tree on March 2020 etc., and we need you to stick to that time table. Without such a plan, we cannot act. From the time we have a concrete, definite reliable and official timeline on our hands, we need 2-3 years.

Thunderbird has a much larger UI surface than Firefox and relies very heavily on all facets of XUL. XUL overlays, on XUL <tree> and all features of XUL. We will need about 3 years to move away from XUL, and 18-24 months to move away from XUL <tree>, given how core it is to our user interface and also to the user experience.

Comment 7

[Dave Townsend [:mossop]]

(In reply to Ben Bucksch (:BenB) from comment #6)
> > Speaking of ahead warning time, my general consideration is that in all communications we've been
> > quite explicit about the intent to remove XUL for quite some time now
> 
> I have been constantly - both outside the TB Council and from within -
> telling everybody that XUL will soon be gone. Every time, a discussion
> unfolds about how Mozilla has announced this since a decade, and that it's
> still years away. But despite all my efforts, nothing moves.
> 
> The lack of a clear, definite and reliable timeline is the very thing that
> has blocked Thunderbird's progress. We need to know: XUL overlay will be
> removed on March 2019, XUL tree on March 2020 etc., and we need you to stick
> to that time table. Without such a plan, we cannot act. From the time we
> have a concrete, definite reliable and official timeline on our hands, we
> need 2-3 years.

I doubt we can give you such an accurate timeline that far out. We certainly intend to give timelines wherever we can but as usual they are always affected by staffing changes etc. That's just the reality of software development.

> Thunderbird has a much larger UI surface than Firefox and relies very
> heavily on all facets of XUL. XUL overlays, on XUL <tree> and all features
> of XUL. We will need about 3 years to move away from XUL, and 18-24 months
> to move away from XUL <tree>, given how core it is to our user interface and
> also to the user experience.

Honestly I doubt you have that long. <tree> might be one of the later ones we remove since as you know there is no direct alternative and we have a bunch of uses in Firefox too, but if we're only finally removing the XUL code in three years time I think something will have gone very wrong.

Comment 8

[Ben Bucksch (:BenB)]

> reality of software development

That's understood. Nonetheless, some planning and time tables are required. Esp. in a case like this where the removal of parts of XUL threatens the very existence of the Thunderbird project. Thunderbird has 25 million users, after all. They love Thunderbird as it is, which puts us TB developers in a very tough spot.

> but if we're only finally removing the XUL code in three years time I think something will have gone very wrong.

Given that some TB Council members think that the XUL removal is years out, we need some *definite* and concrete statements. I understand you, but most others simply don't believe such vague statements. The time table also has a large influence on the solution we choose. So, we really need more concrete information in order to act.

Comment 9

[Dave Townsend [:mossop]]

(In reply to Ben Bucksch (:BenB) from comment #8)
> > reality of software development
> 
> That's understood. Nonetheless, some planning and time tables are required.
> Esp. in a case like this where the removal of parts of XUL threatens the
> very existence of the Thunderbird project. Thunderbird has 25 million users,
> after all. They love Thunderbird as it is, which puts us TB developers in a
> very tough spot.
> 
> > but if we're only finally removing the XUL code in three years time I think something will have gone very wrong.
> 
> Given that some TB Council members think that the XUL removal is years out,
> we need some *definite* and concrete statements. I understand you, but most
> others simply don't believe such vague statements. The time table also has a
> large influence on the solution we choose. So, we really need more concrete
> information in order to act.

We can come up with the broad strokes for when XUL might be completely gone from mozilla-central but the reality is that we will probably remove support for individual elements opportunistically along the way as we remove use of them in Firefox. We'll only understand the specifics of those as we come across them, maybe only being able to give weeks or less notice.

Comment 10

[Ben Bucksch (:BenB)]

That doesn't leave Thunderbird much room to act.

Please coordinate with us.

Comment 11

[:Paolo Amadini]

While I was reviewing a few removals of XUL "tree" features, I took a look at where we are with regards to the removal of the entire widget. At the moment, we're evaluating on a case by case basis whether to keep support for features only used in comm-central, for example bug 1470920 is something that doesn't cost much to keep around and, if nothing changes in the cost evaluation, will be there until we remove the widget.

At some point, however, we'll start tackling the widget more thoroughly, for the purpose of removing the XBL binding and the special support we have to maintain in the layout code. If we remove every "tree" usage from Firefox, or replace it with a simple list, we will just remove the element. Otherwise, we may have to implement new components to support the more complex cases, like hierarchical views and virtual views. However, they would be separate smaller components, for example the one supporting a hierarchical view might be different from the one supporting a virtual view. This will only be driven by the Firefox requirements, in particular for the Library window.

As a timeline for this, I expect the "tree" removal to be situated somewhere in the last 20% of the XBL removal project. I'm looking at this chart right now:

https://bgrins.github.io/xbl-analysis/graph/

If we don't make changes to how the project is staffed, and assume progress is constant, a rough estimate indicates that we will reach this last 20% in February or March 2019. This means that, according to the current release calendar...

https://wiki.mozilla.org/Release_Management/Calendar

...the last version of mozilla-central with platform support for "tree", and maybe for any other XBL binding defined in a ".xml" file, could be:

  Firefox 66

This also means that Firefox 60 will be the last ESR with support for XBL, and again, possibly for a number of other XUL features that we will have removed opportunistically along the way.

While the Firefox 66 target is just my own rough estimate and is quite variable, it is much more certain that Firefox 60 will be the last ESR with this kind of support.

I hope these estimates could help in choosing a technical solution for Thunderbird and SeaMonkey moving forward, whether that includes modernizing the interface, branching the platform code at a point in time, maintaining a set of custom platform patches on a rolling basis, or a combination of these.

Comment 12

[Frank-Rainer Grahl (:frg)]

> whether that includes modernizing the interface,

To modernize the interface you need some basic support to build upon and there is then almost none as I see it. Only removals and nothing added which could support both Thunderbird and SeaMonkey. 

Last post here. Remove it if you want as advocacy.

Comment 13

[Tim Nguyen :ntim]

DevTools have already some React-based trees for their own purposes, I honestly think the most productive way forward for TB and SM would be investigating extending those for comm-central purposes, since it's clear that mozilla-central needs aren't as big as comm-central.

The one found at [0] is probably the most complete one at the moment. It provides Tree-Table like functionality, so a tree view, with different columns if needed, it also provides column headers. It's used in the Accessibility Inspector if you want to see it in action.

I hope that helps.

[0]: https://searchfox.org/mozilla-central/source/devtools/client/shared/components/tree

Comment 14

[Philipp Preißer-Kewisch [:Fallen][☁️📆]]

I agree that Thunderbird needs to move off the xul tree, there are a few alternatives that have been mentioned in multiple locations. Thanks for the timeline, this is really helpful. I realize this would add some maintenance burden, but would it be sensible to postpone the final tree removal until after Firefox 67 ESR? This way we have some more time in case we don't manage to replace it by then. We can evaluate this in detail though in Q1, maybe it doesn't need to be discussed if we've removed the tree usage by then.

Comment 15

[:Paolo Amadini]

The next ESR will be Firefox 68, that's why I think we'll have removed XBL support by then. It may still be possible for Thunderbird to branch from 66 or 67, or maybe branch from 68 and port the changes needed to support the remaining XBL components, if any. I don't think the platform code will have diverged too much between 66 and 68, even though one the purposes of removing XUL is exactly to enable new developments in the platform, so I expect the code to diverge significantly later in 2019.

Comment 16

[Mark Straver]

I think it should be clear by now that any mozilla-central development is only going to be driven by Firefox requirements and on an as-needed basis. It's the only considered application left.

My question as food for thought to the Thunderbird team is therefore: What exactly do you need from mozilla-central and its Quantum developments to drive a mail/news/chat suite? Why aim for 60-ESR to begin with? Why not split off earlier and save yourself the headaches of half-removed XUL and having to work around removals with barely weeks to solve it?

** Do you really /need/ Firefox's new tech? **

Comment 17

[Jorg K (CEST = GMT+2)]

(In reply to Mark Straver from comment #16)
> What exactly do you need from mozilla-central and its Quantum developments
> to drive a mail/news/chat suite?
Quick answer: We need security fixes. We have no capacity to fork and then keep integrating security fixes on a diverging code base. Otherwise we will end up like FossaMail(†) :-( 

> ** Do you really /need/ Firefox's new tech? **
Certainly not all of it but there is no choice.

Comment 18

[Mark Straver]

Not sure what FossaMail has to do with anything - it was discontinued as a public project because of lack of time to support for a larger audience with no source of income to justify the time sunk into such support. Certainly not for lack of security updates (for what little applies to a mail client that doesn't expose much to displayed content) which is something *one man* could do. In fact, the unofficial builds that are still offered to users (with zero support) are secure.

Comment 19

[Philipp Preißer-Kewisch [:Fallen][☁️📆]]

Folks, while this is certainly a discussion worth having (and I believe parts of it have been discussed on tb-planning and other venues), I think this isn't helpful for this specific bug. The Mozilla Platform is a lot more than just XUL, and the dependencies Thunderbird has are too deep to bail at the moment. We are working on reducing dependencies, but with the size of the project it will take a while.

Comment 20

[Magnus Melin [:mkmelin]]

(In reply to :Paolo Amadini [Away] from comment #11)

Thanks Paulo for the timelines! But it seems to me you looked at the wrong column in the release calendar, no?. See below

> If we don't make changes to how the project is staffed, and assume progress
> is constant, a rough estimate indicates that we will reach this last 20% in
> February or March 2019. This means that, according to the current release
> calendar...
> 
> https://wiki.mozilla.org/Release_Management/Calendar
> 
> ...the last version of mozilla-central with platform support for "tree", and
> maybe for any other XBL binding defined in a ".xml" file, could be:
> 
>   Firefox 66

According to the calendar, Firefox 66 would be branching 2018-12-10. 
The February-March estimate corresponds to Firefox 68 branching date 2019-03-18.

Unless of course you intend to backport very aggressively?

Given 68 is the next ESR this would seem to work out ok for Thunderbird, giving us some breathing room until ESR++ to migrating everything over.

Comment 21

[:Paolo Amadini]

(In reply to Magnus Melin from comment #20)
> According to the calendar, Firefox 66 would be branching 2018-12-10. 
> The February-March estimate corresponds to Firefox 68 branching date
> 2019-03-18.

According to the current calendar, anything that lands in mozilla-central before May 15 will be included in Firefox 68, which is the next ESR, so if we landed the tree removal in February or March then trees would not be available there.

That said, we have some notable mozilla-central dependencies for trees as well, in particular the Library window, and I haven't seen lots of activity for the conversion to in-content yet, so it may turn out that we'll have to delay the tree removal plans in order to wait for these other projects to take place, or change the strategy we're using to deal with the removal. I'll keep you posted, and feel free to ask at any time.

Comment 22

[Jorg K (CEST = GMT+2)]

For the record: Ben's experiment: http://benbucksch.github.io/trex/fastlist-test.html

Comment 23

[:Paolo Amadini]

As a quick update for those interested, the XBL removal project proceeded at a constant pace, and has just reached the last 20% of bindings to remove! For the "tree" widget, given we still have various dependencies in Firefox, Victor landed an impressive conversion to Custom Element in bug 1523957, so for the time being Thunderbird will be able to use the element without substantial changes.

Comment 24

[Jorg K (CEST = GMT+2)]

... so for the time being Thunderbird will be able to use the element without substantial changes.

Thanks Paolo and Victor, that work is much appreciated. What's the long-term aim though?

Comment 25

[:Paolo Amadini]

Brian, do you have a general idea of the current plans for dealing with the "tree" element, to answer Jorg's question?

Comment 26

[Tim Nguyen :ntim]

(In reply to :Paolo Amadini from comment #25)

Brian, do you have a general idea of the current plans for dealing with the "tree" element, to answer Jorg's question?

There's pretty much none at the moment. It may be one of the special features of XUL that will be kept unchanged (or will receive few changes), like popups or splitters...

Comment 27

[Brian Grinstead [:bgrins]]

(In reply to Tim Nguyen :ntim from comment #26)

(In reply to :Paolo Amadini from comment #25)

Brian, do you have a general idea of the current plans for dealing with the "tree" element, to answer Jorg's question?

There's pretty much none at the moment. It may be one of the special features of XUL that will be kept unchanged (or will receive few changes), like popups or splitters...

Pretty much this. There've been some investigations into either removing the remaining consumers in Firefox or reusing/building a JS-based replacement but it hasn't been a high priority compared with the rest of the XUL/XBL work. I don't anticipate that happening unless/until all consumers go away from product changes, or the tree implementation is causing problems with other projects (at one point it looked like this would be the case with WebRender but AFAIK that's been resolved). Though if TB has or finds a viable JS-based replacement with a migration path for consumers I'd certainly be interested in helping to get it merged into toolkit.

Comment 28

[M Lopez-Ibanez]

Hopefully relevant: Zotero has implemented their own high-performing replacement to XUL tree:

https://thunderbird.topicbox.com/groups/developers/Tc68d151e839d9e99/html-tree-replacement-for-the-xul-tree