1342054 - Large number of tags causes very bad tags preference UI performance, from autotaging RSS feeds with "Automatically create tags from feed category names"

Status: NEW

(Thunderbird :: Preferences, defect)

Description

[Anthony Carrico]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36

Steps to reproduce:

Name 	Thunderbird
Version 	45.6.0
User Agent 	Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.6.0
Profile Directory 	
(Local drive)
Application Build ID 	20161231151914

This is an Arch Linux build, but I've also tried other recent Thunderbird builds from the Nix package manager with the same profile.

Meta issue: UI is very slow, I've tried many of the Thunderbird troubleshooting ideas out there (safe mode doesn't help, other profiles are very fast, indexing is off).

Specific issue: Preferences->Display->Tags has thousands (estimated) of tags, apparently from autotaging RSS feeds, which I have disabled. I've tried deleting these tags, but it isn't humanly possible to do so from the UI.


Actual results:

I can delete the tags one at a time with the button, but it would take days to do this by hand, especially with the sluggish UI. Thunderbird often uses 100% of one CPU, including while deleting tags in offline mode.

I can't say if these tags are behind the UI sluggishness since there isn't an easy way to delete them.

The <shift>-click or <control>-click tricks for highlighting many list elements doesn't work in the tags dialog.



Expected results:

I could delete tags quickly, ideally by selecting many and pushing the delete button. In a perfect world deleting all these unwanted/unused tags would sort out my UI sluggishness, but that remains to be scene.

BTW I also noticed that the vast majority of the elements in the config editor are of the form:
  mailnews.tags.*.tag
  mailnews.tags.*.ordinal

Comment 1

[alta88]

Yes, this is a problem due to the internal tag design, from when there were 5 hardcoded tags and people wanted custom tags..  Each tag is 2 prefs and the filter dialog is very slow as well due to the menulist being built before drawing the window. Your best workaround is to close Tb and manually delete the lines in the [profile dir]/prefs.js file.

The usefulness of feed tagging very highly depends on the individual feed.

Comment 2

[Anthony Carrico]

Ok. Wow, I deleted 90,000 tags from prefs.js using emacs. I think this may have finally solved months of Thunderbird UI performance troubleshooting. I'll reopen / file new bug if necessary in a few days, but let's tentatively call this one solved. THANK YOU. This issue might deserve an official blog post or something, because none of my searches have put a spotlight on this issue which can really make Thunderbird unusable.

I'd mark this RESOLVED, but not sure which status to use, and also you might want to keep it open for a technical or social fix since Thunderbird clearly has real trouble with thousands of tags. Perhaps even just a strong warning about the bug in the documentation would be a good start.

Comment 3

[alta88]

It should be left open; the entire design needs to be rewritten to handle volume. In addition to the perpetual lost tags in imap problem and several others listed in other bugs.

Comment 4

[alta88]

For the record, in Bug 996690 the original patch omitted autotags from widgets..

Comment 5

[Wayne Mery (:wsmwk)]

Anthony, is tags preference UI the only area that seems slow to you?
For example, getting new RSS is not slow?

Comment 6

[Anthony Carrico]

The entire UI was slow with 90,000ish tags. Drop down menus, scrolling, reaction to clicks, etc. Not just the preferences, but the main application window.

Comment 7

[Anthony Carrico]

Now that I'm back to the default tag set the UI behaves as expected.

Comment 8

[Anthony Carrico]

... three months later, Thunderbird 52.1.1, arch linux:

Things had been smooth, but the upgrade to 52 has had major issues, with random crashes and other weird behavior (see https://bugzilla.mozilla.org/show_bug.cgi?id=1362895#c3). I also noticed sluggish UI, so I checked the tags, and sure enough there were 121 new tags, apparently harvested from feeds. I deleted the tags.

The global autotag preference is off, but tagging is happening, so I checked each feed manually. Some have DO have autotagging enabled, seemingly at random. Incidentally, when I clicked to disable autotagging for the first autotagged feed, Thunderbird crashed, but I think this is due to other problems (see the other bug).

Comment 9

[Wayne Mery (:wsmwk)]

(In reply to Anthony Carrico from comment #8)
> ... three months later, Thunderbird 52.1.1, arch linux:
> 
> ... I also noticed
> sluggish UI, so I checked the tags, and sure enough there were 121 new tags,
> apparently harvested from feeds. I deleted the tags.

What parts of Thunderbird are sluggish??

> The global autotag preference is off, but tagging is happening, so I checked
> each feed manually. Some have DO have autotagging enabled, seemingly at
> random. Incidentally, when I clicked to disable autotagging for the first
> autotagged feed, Thunderbird crashed, but I think this is due to other
> problems (see the other bug).

Please file a different bug for this, and also for the crashing

Comment 10

[Wayne Mery (:wsmwk)]

(In reply to alta88 from comment #3)
> It should be left open; the entire design needs to be rewritten to handle
> volume. 

Ouch!  
Here is another report https://www.reddit.com/r/Thunderbird/comments/4jw6qv/is_it_save_to_remove_unnecessary_tags_from_prefsjs/

Is this auto-tagging of serious value considering the problems it causes?


> In addition to the perpetual lost tags in imap problem and several others listed in other bugs.

Which bugs are you talking about?

Comment 11

[alta88]

First, the defaults for each individual feed and the global setting is off. It is totally up to the user to enable it. If the global setting is enabled, all subsequent newly added feeds will thus be enabled.  (That's what the OP did, it's not at all "seemingly at random").  The updated account settings UI in Tb54 makes it more clear the settings there are for new feeds and not global; all existing feeds are only affected by their own settings in Subscribe.

Anyway, 121 tags will not make anything sluggish, so best look elsewhere.  Also, "The usefulness of feed tagging very highly depends on the individual feed", so it is, and should be, up to the user to decide whether it should be on for a particular feed.  A statement in the Learn More guide about being careful with autotagging of feeds is appropriate (rather than panic).

I don't have a list of bugs handy, but surely you're aware that lost mail tags are a persistent complaint..

The root fix here is twofold:
1) Tags and their metadata should not be 2 prefs per tag, but stored in a json file. It would be reasonable to have a configurable max tags throttle.
2) Widgets should not attempt to create a menu or list with thousands of tags, the original design considered 5 tags. And the management tab in Preferences should be much smarter about loading/be async/yield, etc.

Comment 12

[Wayne Mery (:wsmwk)]

> I don't have a list of bugs handy, but surely you're aware that lost mail tags are a persistent complaint.

There are many possible causes for lost tags. To which are you referring? Dumb imap servers?  Loss after compact? etc. (the ones I have stated have little or nothing to do with RSS tags, so I don't know what you are talking about else I would not have asked)

Comment 13

[alta88]

I meant not only the IMAP case, but everything else to do with tag management, as part of a revamp of the whole thing. Feeds have nothing to do with it other than demonstrating how poor the implementation is.

https://bugzilla.mozilla.org/buglist.cgi?list_id=13609711&v2=Feed%20Reader&short_desc=tag&f4=CP&product=MailNews%20Core&product=Thunderbird&short_desc_type=anywords&f1=OP&f0=OP&resolution=---&classification=Client%20Software&classification=Components&query_format=advanced&j1=OR&f3=CP

Comment 14

[Anthony Carrico]

> Please file a different bug for this, and also for the crashing

I did reference another bug above, where I detailed my problems and solutions for 45->52 issues. I noticed this little reprise of the tagging along the way, and so I added an update here where it is relevant.

Comment 15

[Anthony Carrico]

(In reply to alta88 from comment #11)
> First, the defaults for each individual feed and the global setting is off.
> It is totally up to the user to enable it. If the global setting is enabled,
> all subsequent newly added feeds will thus be enabled.  (That's what the OP
> did, it's not at all "seemingly at random").

While debugging the major 45->52 issues, I noticed new tags had been harvested. Since autotags had been a major source of trouble in the past, I walked the RSS tree and found a few "random" feeds that were still tagging. I made a note here so others will know to manually check all their feeds, and not to count on the global setting when addressing this tag bug for themselves.

This process was very crashy for me, but I believe this was because of other 45->52 upgrade issues which I have solved and noted in the other bug report.

Comment 19

[:aceman]

If the claim here is that all UI is sluggish with many tags, we need to test that (I can try).

So it seems we can distinguish which tags are autogenerated, via tag.ordinal.contains("~AUTOTAG").
Can we have two methods to enumerate all tags? One that returns these autotags and one that doesn't.
It may be that we reference the tag list in many places. You could just make the default enumeration function (getAllTags) return only the real tags so that all UI is happy. Only the single place that uses the autotags would call the new method that returns all tags (or just the autotags actually).

Another solution could be to make nsMsgTagService::GetAllTags faster by returning all tags from a cache, not always re-read all tag pref keys (thousands), sort them and then get the pref values and build the tags and return the list.

If it's not that nsMsgTagService::GetAllTags is slow, but the building of menulists with thousands of items is slow, then you could salvage the parts of bug 996690 and finish them here. Looks like bug 996690 comment 3 wasn't implemented yet.

Also, we could allow deleting multiple tags from the preferences dialog, if we actually want to show the autotags there. We could sort the autotags to the bottom of the list so that user could delete them in one go (select start and end of the range).

Comment 20

[ova]

I reviewed all RSS feeds options and found that ONE, which has the autogeneration of tags option is ON. There where 54000+ auto tags created. After turning that option off my prefs.js shrinkeв down to 200kB from 4.5 mB size.

RESOLVED.

If you have an ideas how to improve it you are welcome.
Thanks a lot from Russia.
My best regards to developers.

Comment 21

[ova]

:aceman notes in Comment 19:

Also, we could allow deleting multiple tags from the preferences dialog, if we actually want to show the autotags there. We could sort the autotags to the bottom of the list so that user could delete them in one go (select start and end range).

I like it. No need to edit prefs.js manually.

Comment 23

[Wayne Mery (:wsmwk)]

https://support.mozilla.org/en-US/questions/1462917#answer-1674267 cites "57k mailnews.tags.* entries in prefs.js"