Open Bug 1446341 Opened 6 years ago Updated 2 years ago

The XUL "tree" widget will be removed from mozilla-central

Categories

(Thunderbird :: Upstream Synchronization, enhancement)

enhancement

Tracking

(Not tracked)

People

(Reporter: Paolo, Unassigned)

References

(Depends on 3 open bugs)

Details

As noted in bug 1446335, as part of the architectural efforts to remove XUL from mozilla-central, we are looking into deleting the "tree" widget entirely.

While this removal will not happen very soon, the eventual plan of record does not involve developing a replacement that provides the same user experience. Instead, each of the current uses of "tree" in mozilla-central will be evaluated independently and replaced with an alternative that works for the specific use case.

Keeping this in mind, I expect that Thunderbird and SeaMonkey will also have to develop alternatives for the portions of their user interface that depend on "tree", or adopt the technology into comm-central.
Thanks for the heads-up. I believe we've already looked into a replacement for the tree widget, but I'd have to find where that was done.
XUL <tree> is completely central to Thunderbird's UI. The XUL <tree> was originally implemented specifically for Thunderbird, and with the needs of Thunderbird in mind. The folder pane and the thread pane are XUL <tree>s, and the thread pane is the most prominent UI of Thunderbird.

It is highly complex and needs:
* Fast enough for up to 1 million entries
* tree: hierarchical entries with expander
* table: multiple columns, sortable, configurable
* the combination of tree and table at the same time
Without XUL <tree>, and without a suitable replacement that at the very minimum does all of that, Thunderbird will simply be dead.

@Everybody: To start, it would be good to compile a list of requirements that the new implementation needs. The above list is just the big picture, but there are many small requirements to be worked out.

This will not be trivial, and won't be done in a weekend. I've already looked, and there's nothing out there that even comes close. This is a 6-12 months project with full time staff, given that so much of Thunderbird hangs off it. We need a suitable replacement that meets all requirements of Thunderbird, *before* XUL <tree> is removed. Otherwise the project is dead.

@Paolo: I understand that this is just a placeholder and you don't have a timeline yet. But this needs a concerted effort to create a replacement, and is a large project on its own. If you want the Thunderbird project to handle this, we'd need 2 years of ahead warning time, and funds to create the replacement, and need to build a development team for it etc..
(In reply to Ben Bucksch (:BenB) from comment #4)
> @Everybody: To start, it would be good to compile a list of requirements
> that the new implementation needs.

I like the approach of taking a bird's eye view at the requirements of Thunderbird as an application, that is followed here and in the linked Thunderbird newsgroup post, and I would go even further in suggesting that a modern user interface for listing messages and threads could have different levels of similarities to the current list with the tiny expander icons and a single overarching scrollbar.

In the end, what this will look like in two to three years is a decision for the Thunderbird community to make, both from a code and interface design perspective.

> @Paolo: I understand that this is just a placeholder and you don't have a
> timeline yet. But this needs a concerted effort to create a replacement, and
> is a large project on its own. If you want the Thunderbird project to handle
> this, we'd need 2 years of ahead warning time, and funds to create the
> replacement, and need to build a development team for it etc..

It looks like the removal of the tree widget has worked as a good conversation starter, probably because it is demonstrably achievable and it does require some dedicated work to replace. However, as you noted in the newsgroup, this is just part of a bigger effort to remove XUL from mozilla-central. Just as an example, bug 1444468 is related to the removal of XUL overlay technology from the platform, which will happen much sooner. This recent thread lists other parts of XUL for which there is an action plan:

https://mail.mozilla.org/pipermail/firefox-dev/2018-March/006240.html

All the little fixes needed as a consequence will likely require more effort overall than a new interface for listing messages. The same long-term planning considerations for Thunderbird also apply to these changes, and I can only anticipate them becoming more pressing as the XUL removal project gains momentum.

Speaking of ahead warning time, my general consideration is that in all communications we've been quite explicit about the intent to remove XUL for quite some time now, and it doesn't certainly come as a surprise, although only more recently this idea started to be an effort concretely led by engineering. In fact, this is the best time to make a Thunderbird counterpart to these concrete engineering plans, and take decisions.

As an engineering choice, we already took the explicit direction of removing the XUL code from mozilla-central entirely, doing this incrementally as we remove features. Given this, I believe that adopting portions of the mozilla-central platform code in comm-central may be the solution to gain the lead time you mention here, and may work for the "tree" widget. This may take different forms, one of which is branching some or all of the top-level folders and maintaining vendor patches. Of course this is just a suggestion from the perspective of someone not deeply involved with comm-central development, and there may be different options available, which Thunderbird, SeaMonkey, and other projects based on mozilla-central can evaluate.
> Speaking of ahead warning time, my general consideration is that in all communications we've been
> quite explicit about the intent to remove XUL for quite some time now

I have been constantly - both outside the TB Council and from within - telling everybody that XUL will soon be gone. Every time, a discussion unfolds about how Mozilla has announced this since a decade, and that it's still years away. But despite all my efforts, nothing moves.

The lack of a clear, definite and reliable timeline is the very thing that has blocked Thunderbird's progress. We need to know: XUL overlay will be removed on March 2019, XUL tree on March 2020 etc., and we need you to stick to that time table. Without such a plan, we cannot act. From the time we have a concrete, definite reliable and official timeline on our hands, we need 2-3 years.

Thunderbird has a much larger UI surface than Firefox and relies very heavily on all facets of XUL. XUL overlays, on XUL <tree> and all features of XUL. We will need about 3 years to move away from XUL, and 18-24 months to move away from XUL <tree>, given how core it is to our user interface and also to the user experience.
(In reply to Ben Bucksch (:BenB) from comment #6)
> > Speaking of ahead warning time, my general consideration is that in all communications we've been
> > quite explicit about the intent to remove XUL for quite some time now
> 
> I have been constantly - both outside the TB Council and from within -
> telling everybody that XUL will soon be gone. Every time, a discussion
> unfolds about how Mozilla has announced this since a decade, and that it's
> still years away. But despite all my efforts, nothing moves.
> 
> The lack of a clear, definite and reliable timeline is the very thing that
> has blocked Thunderbird's progress. We need to know: XUL overlay will be
> removed on March 2019, XUL tree on March 2020 etc., and we need you to stick
> to that time table. Without such a plan, we cannot act. From the time we
> have a concrete, definite reliable and official timeline on our hands, we
> need 2-3 years.

I doubt we can give you such an accurate timeline that far out. We certainly intend to give timelines wherever we can but as usual they are always affected by staffing changes etc. That's just the reality of software development.

> Thunderbird has a much larger UI surface than Firefox and relies very
> heavily on all facets of XUL. XUL overlays, on XUL <tree> and all features
> of XUL. We will need about 3 years to move away from XUL, and 18-24 months
> to move away from XUL <tree>, given how core it is to our user interface and
> also to the user experience.

Honestly I doubt you have that long. <tree> might be one of the later ones we remove since as you know there is no direct alternative and we have a bunch of uses in Firefox too, but if we're only finally removing the XUL code in three years time I think something will have gone very wrong.
> reality of software development

That's understood. Nonetheless, some planning and time tables are required. Esp. in a case like this where the removal of parts of XUL threatens the very existence of the Thunderbird project. Thunderbird has 25 million users, after all. They love Thunderbird as it is, which puts us TB developers in a very tough spot.

> but if we're only finally removing the XUL code in three years time I think something will have gone very wrong.

Given that some TB Council members think that the XUL removal is years out, we need some *definite* and concrete statements. I understand you, but most others simply don't believe such vague statements. The time table also has a large influence on the solution we choose. So, we really need more concrete information in order to act.
(In reply to Ben Bucksch (:BenB) from comment #8)
> > reality of software development
> 
> That's understood. Nonetheless, some planning and time tables are required.
> Esp. in a case like this where the removal of parts of XUL threatens the
> very existence of the Thunderbird project. Thunderbird has 25 million users,
> after all. They love Thunderbird as it is, which puts us TB developers in a
> very tough spot.
> 
> > but if we're only finally removing the XUL code in three years time I think something will have gone very wrong.
> 
> Given that some TB Council members think that the XUL removal is years out,
> we need some *definite* and concrete statements. I understand you, but most
> others simply don't believe such vague statements. The time table also has a
> large influence on the solution we choose. So, we really need more concrete
> information in order to act.

We can come up with the broad strokes for when XUL might be completely gone from mozilla-central but the reality is that we will probably remove support for individual elements opportunistically along the way as we remove use of them in Firefox. We'll only understand the specifics of those as we come across them, maybe only being able to give weeks or less notice.
That doesn't leave Thunderbird much room to act.

Please coordinate with us.
Depends on: 1470920
Depends on: 1430374
Depends on: 1471542
While I was reviewing a few removals of XUL "tree" features, I took a look at where we are with regards to the removal of the entire widget. At the moment, we're evaluating on a case by case basis whether to keep support for features only used in comm-central, for example bug 1470920 is something that doesn't cost much to keep around and, if nothing changes in the cost evaluation, will be there until we remove the widget.

At some point, however, we'll start tackling the widget more thoroughly, for the purpose of removing the XBL binding and the special support we have to maintain in the layout code. If we remove every "tree" usage from Firefox, or replace it with a simple list, we will just remove the element. Otherwise, we may have to implement new components to support the more complex cases, like hierarchical views and virtual views. However, they would be separate smaller components, for example the one supporting a hierarchical view might be different from the one supporting a virtual view. This will only be driven by the Firefox requirements, in particular for the Library window.

As a timeline for this, I expect the "tree" removal to be situated somewhere in the last 20% of the XBL removal project. I'm looking at this chart right now:

https://bgrins.github.io/xbl-analysis/graph/

If we don't make changes to how the project is staffed, and assume progress is constant, a rough estimate indicates that we will reach this last 20% in February or March 2019. This means that, according to the current release calendar...

https://wiki.mozilla.org/Release_Management/Calendar

...the last version of mozilla-central with platform support for "tree", and maybe for any other XBL binding defined in a ".xml" file, could be:

  Firefox 66

This also means that Firefox 60 will be the last ESR with support for XBL, and again, possibly for a number of other XUL features that we will have removed opportunistically along the way.

While the Firefox 66 target is just my own rough estimate and is quite variable, it is much more certain that Firefox 60 will be the last ESR with this kind of support.

I hope these estimates could help in choosing a technical solution for Thunderbird and SeaMonkey moving forward, whether that includes modernizing the interface, branching the platform code at a point in time, maintaining a set of custom platform patches on a rolling basis, or a combination of these.
> whether that includes modernizing the interface,

To modernize the interface you need some basic support to build upon and there is then almost none as I see it. Only removals and nothing added which could support both Thunderbird and SeaMonkey. 

Last post here. Remove it if you want as advocacy.
DevTools have already some React-based trees for their own purposes, I honestly think the most productive way forward for TB and SM would be investigating extending those for comm-central purposes, since it's clear that mozilla-central needs aren't as big as comm-central.

The one found at [0] is probably the most complete one at the moment. It provides Tree-Table like functionality, so a tree view, with different columns if needed, it also provides column headers. It's used in the Accessibility Inspector if you want to see it in action.

I hope that helps.

[0]: https://searchfox.org/mozilla-central/source/devtools/client/shared/components/tree
I agree that Thunderbird needs to move off the xul tree, there are a few alternatives that have been mentioned in multiple locations. Thanks for the timeline, this is really helpful. I realize this would add some maintenance burden, but would it be sensible to postpone the final tree removal until after Firefox 67 ESR? This way we have some more time in case we don't manage to replace it by then. We can evaluate this in detail though in Q1, maybe it doesn't need to be discussed if we've removed the tree usage by then.
Flags: needinfo?(paolo.mozmail)
The next ESR will be Firefox 68, that's why I think we'll have removed XBL support by then. It may still be possible for Thunderbird to branch from 66 or 67, or maybe branch from 68 and port the changes needed to support the remaining XBL components, if any. I don't think the platform code will have diverged too much between 66 and 68, even though one the purposes of removing XUL is exactly to enable new developments in the platform, so I expect the code to diverge significantly later in 2019.
Flags: needinfo?(paolo.mozmail)
(In reply to :Paolo Amadini [Away] from comment #11)

Thanks Paulo for the timelines! But it seems to me you looked at the wrong column in the release calendar, no?. See below

> If we don't make changes to how the project is staffed, and assume progress
> is constant, a rough estimate indicates that we will reach this last 20% in
> February or March 2019. This means that, according to the current release
> calendar...
> 
> https://wiki.mozilla.org/Release_Management/Calendar
> 
> ...the last version of mozilla-central with platform support for "tree", and
> maybe for any other XBL binding defined in a ".xml" file, could be:
> 
>   Firefox 66

According to the calendar, Firefox 66 would be branching 2018-12-10. 
The February-March estimate corresponds to Firefox 68 branching date 2019-03-18.

Unless of course you intend to backport very aggressively?

Given 68 is the next ESR this would seem to work out ok for Thunderbird, giving us some breathing room until ESR++ to migrating everything over.
(In reply to Magnus Melin from comment #20)
> According to the calendar, Firefox 66 would be branching 2018-12-10. 
> The February-March estimate corresponds to Firefox 68 branching date
> 2019-03-18.

According to the current calendar, anything that lands in mozilla-central before May 15 will be included in Firefox 68, which is the next ESR, so if we landed the tree removal in February or March then trees would not be available there.

That said, we have some notable mozilla-central dependencies for trees as well, in particular the Library window, and I haven't seen lots of activity for the conversion to in-content yet, so it may turn out that we'll have to delay the tree removal plans in order to wait for these other projects to take place, or change the strategy we're using to deal with the removal. I'll keep you posted, and feel free to ask at any time.
Depends on: 1508141
Depends on: 1508142
Depends on: 1508143
Depends on: 1508146
Depends on: 1508151
Depends on: 1508165
Depends on: 1508169

As a quick update for those interested, the XBL removal project proceeded at a constant pace, and has just reached the last 20% of bindings to remove! For the "tree" widget, given we still have various dependencies in Firefox, Victor landed an impressive conversion to Custom Element in bug 1523957, so for the time being Thunderbird will be able to use the element without substantial changes.

... so for the time being Thunderbird will be able to use the element without substantial changes.

Thanks Paolo and Victor, that work is much appreciated. What's the long-term aim though?

Brian, do you have a general idea of the current plans for dealing with the "tree" element, to answer Jorg's question?

Flags: needinfo?(bgrinstead)

(In reply to :Paolo Amadini from comment #25)

Brian, do you have a general idea of the current plans for dealing with the "tree" element, to answer Jorg's question?

There's pretty much none at the moment. It may be one of the special features of XUL that will be kept unchanged (or will receive few changes), like popups or splitters...

(In reply to Tim Nguyen :ntim from comment #26)

(In reply to :Paolo Amadini from comment #25)

Brian, do you have a general idea of the current plans for dealing with the "tree" element, to answer Jorg's question?

There's pretty much none at the moment. It may be one of the special features of XUL that will be kept unchanged (or will receive few changes), like popups or splitters...

Pretty much this. There've been some investigations into either removing the remaining consumers in Firefox or reusing/building a JS-based replacement but it hasn't been a high priority compared with the rest of the XUL/XBL work. I don't anticipate that happening unless/until all consumers go away from product changes, or the tree implementation is causing problems with other projects (at one point it looked like this would be the case with WebRender but AFAIK that's been resolved). Though if TB has or finds a viable JS-based replacement with a migration path for consumers I'd certainly be interested in helping to get it merged into toolkit.

Flags: needinfo?(bgrinstead)
See Also: → 1619697
Component: General → Upstream Synchronization

Hopefully relevant: Zotero has implemented their own high-performing replacement to XUL tree:

https://thunderbird.topicbox.com/groups/developers/Tc68d151e839d9e99/html-tree-replacement-for-the-xul-tree

Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.