1933365 - popstate.dat is not portable between OSes

Status: RESOLVED FIXED

(MailNews Core :: General, defect)

Description

[Ben Campbell]

popstate.dat holds the list of messages we know about on a POP3 server.
The Pop3Client._loadUidlState() method loads them in, but the parsing relies on very specific line endings which differ by OS.

https://searchfox.org/comm-central/source/mailnews/local/src/Pop3Client.sys.mjs#475

So if you copied your profile from a linux machine to windows, it wouldn't be able to read in the UIDLs - it'd be trying to split lines on CRLF, but the file would only have LFs.

Not 100% sure how portable profiles are (or should be) between machines, but this seems needlessly platform-specific and easy enough to fix up (look for LF line endings and just trim any CRs you see).

Comment 1

[ISHIKAWA, Chiaki]

Ben,

It is a good catch.

I once tried to share a directory between Linux and Window OS and gave up due to the various incompatibility issues.
I THINK in the ancient versions even folders (read MBox format) behaved differently regarding line ending issue.

Please fix this if you can.

There will be naive users who THINK that the mail data would be portable and they can share a single copy of profile (assuming ONLY one client on different OS access it at a time) and get bitten badly like I was many years ago.

So removal of any incompatibilities in sight would be useful.

HOWEVER, I am aware of other incompatibility issues such as, hmm, the following came to my mind immediately.

• when save a message as a file, TB seems to use the content of the "Subject:" as the file path name by default.
  Naturally, there will be characters in "Subject:" which are not legitimate as part of file pathname under the particular OS TB runs on.
  TB replaces them with other characters such as "_" IIRC. (Or "-". It could be OS-specific!)

I found the replacement is done in an OS-specific manner (naturally the characters that are not legal as part of a file pathname,
may be legal in other OS.)

But this obviously causes the incompatibility issue if the saved file resides in a shared file system used by different OSs.
(And for that matter, the hosting OS that provides the shared filesystem may NOT understand the pathname.)
So the produced path name should be as OS-agnostic as possible.
I would rather see the union of illegal characters on popular OS (Windows, Linux, and MacOS) to be treated
as the target of such replacement characters instead of OS-specific character set on individual OS.
In today's world, if we cover, POSIX and Windows in a largest-common-denominator manner, I think we are fine.

I noticed this issue while I try to move the saved messages across OSs via shared folder many years ago.
And as a matter of fact, it has been a constant hassle since I introduced TrueNAS file server to store old e-mail messages and
an online TB profile backup using OwnCloud. TrueNAS runs on FreeBSD and some files crreated by TB in the past
caused illegal path name issues on the server and eventually on the windows client, too. It can bite in both ways.
When I was flooded with OwnCloud complaining something about the pathname issues was when I realized that saved messages use the content of "Subject:" line as pathname and the replacement algorithm is not robust in the presence of shared remote file system that is used by different OSs (in my case, Windows and linux basically. I test C-C TB patches under linux and sometimes the test files are stored in a drecitory shared under OwnCloud.)
Funny, after I switched to NextCloud, I am not sure if I have noticed the issue.

I should file the isseu above. Maybe I have already, but not sure.

Because more and more people may begin to use such shared file service at home and on cloud,.
Any incompatibility across different OSs should be removed IMHO.

If different OSs proliferate, that will be an issue, but as I see it, file pathname won't be so free after all even if new OS appears [and developers will stick to the current practice more or less], and
covering Windows and POSIX (linux, MacOS, FreeBSD, etc.) goes a long way.

Comment 3

[Thomas E. Horner]

Why 'wontfix'?
It shouldn't be a complicated thing to use standardized line endings would and make the life of users and admins a lot easier. Why's this OSS not multi platform capable from the beginning?!

Comment 4

[Ben Campbell]

It's just marked wontfix for the 128esr version, which is already released.

The intention is to fix it.
(or, even better, get rid of pop3state.dat altogether)

Comment 5

[Thomas E. Horner]

Fantastic, thanks for the clarification 😎👍🏻

Having sqlite databases already, that might be a simple alternative to store already-seen message ids.
Parsing all local messages for message-ids might be too I/O and CPU intensive.

Comment 6

[Hartmut Welpmann [:welpy-cw]]

Attachment: Bug 1933365 - Make parsing of popstate.dat independent of specific line endings. r=#thunderbird-back-end-reviewers

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/4ee1acf937db
Make parsing of popstate.dat independent of specific line endings. r=mkmelin