Closed Bug 992879 Opened 8 years ago Closed 8 years ago

Folders created with a colon (:) or star (*) in Tb24 on linux are duplicated with hash names on startup with Tb trunk or earlier, such local folders/feeds do no longer work

Categories

(MailNews Core :: Backend, defect)

x86_64
Linux
defect
Not set
critical

Tracking

(thunderbird32 fixed, thunderbird_esr3132+ fixed)

RESOLVED FIXED
Thunderbird 33.0
Tracking Status
thunderbird32 --- fixed
thunderbird_esr31 32+ fixed

People

(Reporter: kairo, Assigned: alta88)

References

(Blocks 1 open bug)

Details

(Keywords: dataloss, regression, Whiteboard: [regression:TB29])

Attachments

(1 file)

In the last week or two, I found that on my SeaMonkey nightly builds, all feeds that contain a colon (:) in their name (what appears as folder name in the UI) do not work any more, and their names appear doubled in the folder view.
I'm seeing a few of those in the error console when I launch SeaMonkey, not sure if they are related:

Timestamp: 08.04.2014 13:42:45
Error: NS_ERROR_FILE_IS_DIRECTORY: Component returned failure code: 0x8052000d (NS_ERROR_FILE_IS_DIRECTORY) [nsIMsgLocalMailFolder.addMessage]
Source file: chrome://messenger-newsblog/content/FeedItem.js
Line: 349
On latest trunk, renaming a feed folder to one with a colon wfm, ie new messages are received fine afterwards.

There may be something wrong with your profile folders, such as incompletely deleted folders or such, that don't appear in folderpane but do on disk, or are missing a .msf or need it rebuilt.
This happened to me in both my SeaMonkey installations on different computers that had long-existing feeds with colons in the name. It also doesn't work to rename those folders to something without colons after the fact.
Severity: critical → major
OK, I see that I have actual folder and file names on disk with those colons (and the star for Diapspora*) and what we're actually using is names that look like "Status Board0bae8b9a" when the folder name is "Status Board: Team" (just as the on-disk file I have). For some reason, we then create an empty directory on disk named "Status Board0bae8b9a" instead of a file, which also sounds fishy.
Just noticed this as well for all of my github subscriptions which include ":master" in their name.

STR:
* run 24.6.0 on linux with a new profile
* create a feed account and subscribe to a github feed (for example https://github.com/vvk-ehk/evalimine/commits/master.atom)
* shut down Thunderbird
* notice that in your profile folder you have "Recent Commits to evalimine:master" and "Recent Commits to evalimine:master.msf" files
* start the same profile with 31.0 beta build 20140704014635
* notice that an empty "Recent Commits to evaliminef63ada25" folder and "Recent Commits to evaliminef63ada25.msf" file appear in your profile
* notice that you cannot see previously retrieved items in either of the folders named "Recent Commits to evalimine:master" that appear in folderpane

This needs a regression window, if I can find the time I'll try to pinpoint this. I didn't see the message mentioned in comment 1. I also think that new feeds added with colons do work (although filenames are without colons now), just old feeds stop working.
Keywords: regression
Ok, I'm confused. The first cc nightly that has this problem is from 0322. But nothing from http://hg.mozilla.org/comm-central/pushloghtml?fromchange=7256839a4e30&tochange=15d8e9787b16 nor http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=199e65efd08b&tochange=c4046fbf9e9d looks suspicious to me.

Can someone confirm that 0321 works and 0322 is indeed the first to have this issue?
This problem is not confined to feed folders.  If you create a folder such as Test:Colon in a Tb24 pop account, upon restarting with trunk, the same hashed folders are created on startup folder processing.

However, it doesn't seem to happen on win (checked on winxp), as a folder with a colon is already created with a hashed disk name in Tb24 and starting the profile with trunk has no apparent deleterious effect.

This is borderline data lossy, as to recover Tb needs to be shut down, the newly created hash folder .msf and hash directory need to be deleted, and the existing folder/.msf with the colon need to be renamed to recover.  Of course, this will break things that rely on folder URI; a folder repair may fix some things.  For feeds, it will mean a resubscribe to the new folder name.
Component: Feed Reader → Backend
Summary: Feeds with colons (:) in their names do not work → Folders created with a colon (:) in Tb24 on linux are duplicated with hash names on startup with Tb trunk or earlier, breaking feeds
Does bug 66763 look dodgy?
(In reply to alta88 from comment #8)
> Does bug 66763 look dodgy?

No. https://hg.mozilla.org/comm-central/rev/3320c766cf5b#l5.40 appends (1), (2) and similar not strings like f63ada25.
Yes, that's the one.

This was a bane of feeds for a long time, thus this:
http://mxr.mozilla.org/comm-central/source/mailnews/extensions/newsblog/content/FeedUtils.jsm#626

was implemented. I wonder if they check strings for nonprintable chars and reserved file names..

Tb's attempt at correcting (now newly) illegal filename chars is quite a fail, and it probably shouldn't rewrite existing filenames at all.  Looks like it's done on db file open:
http://mxr.mozilla.org/comm-central/source/mailnews/base/util/nsMsgDBFolder.cpp#909
actually, it's here:
http://mxr.mozilla.org/comm-central/source/mailnews/base/util/nsMsgUtils.cpp#550

a possible fix is to not call NS_MsgHashIfNecessary(pathPiece) #ifdef XP_UNIX and assume the folder uri has already been hashed previously, in the same manner as for XP_MACOSX.
Attached patch colon.patchSplinter Review
this fixes it.  undoubtedly there is rework here, but that could be a long and fragile thread to start pulling.
Assignee: nobody → alta88
Attachment #8452449 - Flags: review?(standard8)
this will be seen as dataloss for any local folder with win special chars on linux, or feeds only folders on mac.
Blocks: 960749
Severity: major → critical
Keywords: dataloss
Comment on attachment 8452449 [details] [diff] [review]
colon.patch

Looks good r=Standard8. Do we need to get this onto 31 as well?
Attachment #8452449 - Flags: review?(standard8) → review+
Comment on attachment 8452449 [details] [diff] [review]
colon.patch

[Approval Request Comment]
Regression caused by (bug #): bug 960749
User impact if declined: as above
Testing completed (on c-c, etc.):  locally
Risk to taking this patch (and alternatives if risky): very low.
Attachment #8452449 - Flags: approval-comm-beta?
Attachment #8452449 - Flags: approval-comm-aurora?
(In reply to Mark Banner (:standard8) from comment #15)
> Comment on attachment 8452449 [details] [diff] [review]
> colon.patch
> 
> Looks good r=Standard8. Do we need to get this onto 31 as well?

yes.  do you auto checkin approved branch uplifts or does the patch author have to follow those up with checkin-needed?  because there are a few others that haven't gone in yet.
https://hg.mozilla.org/comm-central/rev/f07fbe5658ae
Status: NEW → RESOLVED
Closed: 8 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 33.0
Attachment #8452449 - Flags: approval-comm-beta?
Attachment #8452449 - Flags: approval-comm-beta+
Attachment #8452449 - Flags: approval-comm-aurora?
Attachment #8452449 - Flags: approval-comm-aurora+
Comment on attachment 8452449 [details] [diff] [review]
colon.patch

[Approval Request Comment]
Regression caused by (bug #): bug 960749
User impact if declined: local folders with e.g. colon and star in name do not work

This had approval for aurora+beta but apparently never got checked in there.
Attachment #8452449 - Flags: approval-comm-esr31?
Duplicate of this bug: 1047788
Summary: Folders created with a colon (:) in Tb24 on linux are duplicated with hash names on startup with Tb trunk or earlier, breaking feeds → Folders created with a colon (:) or star (*) in Tb24 on linux are duplicated with hash names on startup with Tb trunk or earlier, breaking feeds
Summary: Folders created with a colon (:) or star (*) in Tb24 on linux are duplicated with hash names on startup with Tb trunk or earlier, breaking feeds → Folders created with a colon (:) or star (*) in Tb24 on linux are duplicated with hash names on startup with Tb trunk or earlier, such local folders/feeds do no longer work
Duplicate of this bug: 1048056
Duplicate of this bug: 1048059
Comment on attachment 8452449 [details] [diff] [review]
colon.patch

This is now on aurora, so cancelling the aurora request.
Attachment #8452449 - Flags: approval-comm-aurora+
Whiteboard: [regression:TB29]
Attachment #8452449 - Flags: approval-comm-esr31? → approval-comm-esr31+
You need to log in before you can comment on or make changes to this bug.