Open Bug 1802145 Opened 3 years ago Updated 1 month ago

Restore movemail / unix mailspool support in Thunderbird

Categories

(MailNews Core :: Movemail, task, P1)

Product:
MailNews Core
Component:
Movemail
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: thomas8, Assigned: mozilla)

References

(Blocks 1 open bug)

Details

+++ This bug was initially created as a clone of Bug #1727931 +++

Let's use this bug for a clean start to restore movemail support in Thunderbird (courtesy of QA). It's up to devs if they want to do the work right here or use this as a meta bug. This bug is reserved for technical implementation - no advocacy please (which is already sufficiently present on Bug 1625741 Comment 15 ff and Bug 1727931).

(In reply to Ryan Lee Sipes from bug 1625741 comment #87)

Pulling this out was a mistake. I'm sorry that this decision was made and I'm here to remedy it. We're going to bring Movemail back. If someone here hits us with a patch, it will go faster. - Thunderbird Product Manager

(In reply to Alessandro Castellani [:aleca] from bug 1625741 comment #88)

Let's see if we can simply back out this [bug 1625741] without too much trouble, otherwise we can create a patch that reverts the changes and uplift it to 102.
Gently pinging Ben and Geoff to see how doable this is.

(In reply to Geoff Lankow (:darktrojan) from bug 1625741 comment #95)

(In reply to Alessandro Castellani [:aleca] from comment #88)

Let's see if we can simply back out this without too much trouble, otherwise we can create a patch that reverts the changes and uplift it to 102.

No a simple backout won't work, there's quite a bit of front-end things that would need fixing up plus the XPCOM registration has changed. Also the strings are gone.

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0

Steps to reproduce:

Open Thunderbird and fetch mail

Actual results:

After the latest update to Thunderbird 91 my localhost mailbox disappeared and I couldn't fetch internal linux system mail anymore. It seems movemail has been removed from Thunderbird, why????!!!! Stripping anything that isn't used much will not help Mozilla in the long run.

Expected results:

Movemail should have stayed, it is still used by many Linux users. But it is also very annoying that the localhost mail folder containing old system mails was dropped, it should still be visible since it is a mailbox

(In reply to Thomas D. (:thomas8) from comment #0)

+++ This bug was initially created as a clone of Bug #1727931 +++

Let's use this bug for a clean start to restore movemail support in Thunderbird (courtesy of QA). It's up to devs if they want to do the work right here or use this as a meta bug. ...

(In reply to Ryan Lee Sipes from bug 1625741 comment #87)

Pulling this out was a mistake. I'm sorry that this decision was made and I'm here to remedy it. We're going to bring Movemail back.

Blocks: 1727931
Type: defect → task
Component: Account Manager → Movemail
Priority: -- → P3
See Also: → 1625741

Having engaged in thread 1625741 which states to continue on thread 1727931 I just commented there after searching and finding the reinstatement patch in Betterbird's repo. That patch should provide at least some value either as a copy-paste shortcut or at least for reference. My comment was probably more appropriate for this thread though (implementation, not advocation). Anyway, am linking it here & quoting it below.

(In reply to plelivel from comment #23)

That is good news. As the Thunderbird fork 'Betterbird' already restored Movemail successfully in a recent build, I think it should be possible to merge that code with Thunderbird quite easy.

I had a quick look at the Betterbird Github repo and found the following:

... also several Windows (cmd) scripts for reinstating translations ...

Those scripts provide additional strings during repacking for the various languages. Movemail strings were removed from the L10N repo in a general purge of no-longer-used strings, for example here:
https://hg.mozilla.org/l10n-central/de/rev/db3369a62402dce48cdfbc61add35a34270124b5#l83.12
https://hg.mozilla.org/l10n-central/it/rev/4a0ad63858a5bbc11538a0e06d3e1dd9304c845c#l83.12
https://hg.mozilla.org/l10n-central/ja/rev/78521dfda581a3ab4b2b9e8472950eeea17518dc#l20.12

Is there any roadmap or report somewhere about the current fix of this bug ?

Severity: -- → S1
tracking-firefox129: --- → ?
tracking-firefox129: ? → ---
tracking-firefox130: --- → ?
tracking-firefox130: ? → ---
Priority: P3 → P1

Honestly, if I am involved in building a new support I would like to include SSH support and have it modular enough to connect to arbitrary users for maximum flexibility.

What do you think?

I haven't been able to envision what use case[s] might require SSH, and what do mean by β€œconnect to arbitrary users”? (Perhaps, I'm less familiar with this functionality than you are.)

(In reply to Mr. Beedell, Roke Julian Lockhart (RJLB) from comment #10)

I haven't been able to envision what use case[s] might require SSH, and what do mean by "connect to arbitrary users"? (Perhaps, I'm less familiar with this functionality than you are.)

Well, as far as I understand it, one of the main use cases for traditional Unix mail is status reports from cron jobs as well as certain other local system logs.

Hence I consider it beneficial to be able to connect to inboxes stored on remote servers. And while you are at it, if you have sufficient permissions in some way, I do not see a strong reason not to enable that also for users who do not have a normal login path, for example via sudo -u $username depending on the exact implementation.

Can you briefly describe your intended setup, please.

In retrospect, that's very reasonable. Having it restricted to the local machine would be rather like solely being able to read mail whilst locally logged-into one's mail server.

Can you briefly describe your intended setup, please.

I solely wanted it because something unexpectedly posted mail to my local spool, so I wanted to read it.

This sounds like feature creep happening. If KISS gets it done sooner, I vote for that. The capability to read local mail spool files has been with unixes for 40+ years.

(In reply to greg bell from comment #13)

This sounds like feature creep happening. If KISS gets it done sooner, I vote for that. The capability to read local mail spool files has been with unixes for 40+ years.

Where do you get your E-Mails from?

It looks like the Thunderbird core devs really didn't want to keep the old code sticking around and when rebuilding it better why keep it limited to the old constraints?

I agree with Greg. Just get movemail back.

To answer the mail provenance question: I admit that my use-case is weird but here it is: I download my google mail with a cron-driven IMAP client (getmail), pipe it through procmail for sorting and spam filtering and finally a subset that needs attention lands in the Unix mailbox. I do not use thunderbird's IMAP because I do not want to sync the mail in my thunderbird with that on the server, i.e. I want to delete messages on the local machine without deleting them on the google server.

But here's another real-world scenario used at a university where I worked some years ago: The Unix mailboxes of all employees were in /var/spool/mail on the mail server of the university. This directory was exported over NFS by the mail server and was mounted by each hard-drive-less machine of each employee.

So it is not only mail produced by local daemons that ends up in the Unix mailboxes.

To all the absolute experts on that subject out here, why was macOS explicitly blocked in the past in Thunderbird and still is with Betterbird for that feature?

(In reply to Max Emig from comment #16)

To all the absolute experts on that subject out here, why was macOS explicitly blocked in the past in Thunderbird and still is with Betterbird for that feature?

Is the info in https://mzl.la/4soY18A => Bug 365879 - Don't ship movemail.rdf on the Mac ?

I was informed it did ship at some point for Mac OS X, yet I still have no clue what the original gating was for.

This sounds like feature creep happening

If I end up shipping a patch, it will be a clean Unix Mail implementation and not a sloppy revert of bug 1625741. The code wasn't in a good shape back then including but not limited to bad UX and Betterbird only did the bare minimum in their tree and did not even release the full patch for ESR 140 as they had claimed.

Flags: needinfo?(alessandro)

There was a new needinfo request, but nothing discernible in this thread that required any new information.

If you have a question, please ask it here.

Flags: needinfo?(alessandro)

Please don't remove my needinfo. I manage the Thunderbird team, trying to put eyes back on this bug to get movement.

Flags: needinfo?(alessandro)

Thanks Ryan, I think Max removed it because him and I have been talking on Matrix about this for the past few days.
I'm fully aware of this effort πŸ’ͺ

Flags: needinfo?(alessandro)
Assignee: nobody → mozilla
Status: NEW → ASSIGNED

Excellent. Alright, the removal is reasonable then. Carry on! πŸ˜…

Excellent. Alright, the removal is reasonable then. Carry on! πŸ˜…
To be honest, I did not recognise you right away and had never seen this use of the needinfo-flag without a note before.

Indeed, I am now confident enough in my draft implementation, that I assigned this to myself.

Compared to the old one, it will offer the ability to watch the spooler and get a quasi-push experience with less than 2 seconds of delay and a basic ability to "work in mailbox" which means not pruning it after polling and syncing read-state two-ways using the simple but well-supported Status: R.

Any comments on that?

I think it is a good idea to no longer call this movemail, as that is an ancient tool that hardly anyone actually uses anymore as well as to separate the removed implementation from the new one for future debugging purposes.

I picked the term Unix Mail including #ifdef MOZ_UNIXMAIL which seems to serve the purpose well, but if some of the purists want to come up with another one, there is still time to change that before my first draft patch submission.

Regarding my choice of languages:

JS: The mbox parser is written in JS with RegExp logic loosely based* on the parsing logic in Mutt.

C++: Apart from some glue, the spool file watcher is the only new logic purposely written in C++ as it has the narrow task of firing on change of mtime and/or file size lightly inspired by rsync.

Rust: I considered using Rust more, but ended up using it only for the mentioned syncing of the read-state.

*no derivative code, just as a logic reference as there is unfortunately no definitive standard covering everything widely used.

It looks like the way to go will be a WIP patch with separate build artefacts and further refining with your feedback until its good enough for the main tree.

Main limitations:

  • Moving profiles across machines may not be perfect yet
  • Only the default mbox paths are supported
  • No import from e-mails back into the spooler, even when moving them into the folder
  • No unit tests

Please leave a thumbs up if you are willing to test drive that then.

I am honestly a bit conflicted. It is a big feature, 1000 LOC of code including everything, very vocal minority but few people actually showing up and volunteering to do something.

I believe we can get this new version into mainline Thunderbird eventually, but I could really use 1-2 people familiar with how it is supposed to work (not how it worked in the past) as testers with potentially some light pre code review.

I would like to host a questionnaire on which parts of the feature people would actually want to use if they existed, but I doubt we can reach enough people to get a meaningful result back out.

Yes, I did in fact not release my WIP-patch yet, because it needs a little more work before I would want to run it on other people's main computers.

And I really don't want to end up as a bad example in the so-called LunԁukС Јоurnal* or something.

*yes, this is a crime against Unicode on purpose

I would see the following functionality:

AFAIK there are two mailbox formats: the mbox and maildir. So a first step when setting up a "movemail" account is to specify the format. Another setup option that I see (but it is optional) is to say if the mails in the Unix mailbox should be moved to the global inbox or to a separate inbox (something like the "Local folders"). But I can live without this option and moving mails to the global inbox is fine for me.

Thunderbird should monitor the Unix mailbox (the unique mbox file or the maildir) with some mechanism. AFAIK inotify does not work for NFS but I may be wrong. I don't know how other network file systems (samba, Andrew, Ceph, etc) or "foreign" file systems (an NTFS or VFAT mounted in Linux for example) work with respect to inotify and its equivalents in the Windows world. Polling is the least elegant but should work in all cases.

As soon as Thundebird detects a new complete mail it should move it to the designated inbox (the global one in the simple setup) and issue a desktop notification. The desktop notification lacked in the old movemail and I would have liked to have it. If you see use-cases where the desktop notification is undesirable, then please make it an option in the movemail account setup, but do not remove the possibility to have a desktop notification.

I suppose this "complete mail" requirement is a tricky one. Thunderbird should not start moving the mail as soon as it detects a mbox file modification (an IN_MODIFY event in inotify parlance). On the other hand if it monitors an mbox close event (an IN_CLOSE_WRITE event in inotify parlance) then it risks waiting forever if the mbox-modifying agent keeps the mbox file open in append mode. Polling avoids these issues.

As there is resource contention between movemail and mbox appending agents, some file locking mechanisms should be used but I don't know how well these work on network file systems.

(In reply to Sorin Manolache from comment #15)

To answer the mail provenance question: I admit that my use-case is weird but here it is: I download my google mail with a cron-driven IMAP client (getmail), pipe it through procmail for sorting and spam filtering and finally a subset that needs attention lands in the Unix mailbox. I do not use thunderbird's IMAP because I do not want to sync the mail in my thunderbird with that on the server, i.e. I want to delete messages on the local machine without deleting them on the google server.

FYI, you can obtain the same behavior by asking Thunderbird to copy all you Imap mails to local folders ;-)

Of course, I still support the UNIX mail spool support return anyway :-)

AFAIK there are two mailbox formats: the mbox and maildir.

The maildir is technically supported and even our default storage format, but I am pretty sure it diverged too far to be actually useful in such a setup.

AFAIK inotify does not work for NFS but I may be wrong.

I am considering automatic fallback options, but you are free to disable the push and set any polling interval you like the standard way.

As soon as Thundebird detects a new complete mail it should move it to the designated inbox (the global one in the simple setup)

In my implementation, it is effectively a cross between a local folder and an account, but internally it is an account type and will show up by default as its own inbox as $username@$hostname, but can be renamed later.

I suppose this "complete mail" requirement is a tricky one. Thunderbird should not start moving the mail as soon as it detects a mbox file modification (an IN_MODIFY event in inotify parlance).

I am implementing the standard locking mechanism, I am not even sure whether there is a client widely disregarding that.

As there is resource contention between movemail and mbox appending agents, some file locking mechanisms should be used but I don't know how well these work on network file systems.

I will make sure it works in the way FreeBSD describe it, but if Mutt can't handle it sanely, chances are neither will we.

Thank you so much for your input. 🀩

finally a subset that needs attention lands in the Unix mailbox. I do not use thunderbird's IMAP because I do not want to sync the mail in my thunderbird with that on the server, i.e. I want to delete messages on the local machine without deleting them on the google server.

Would you use pruning or syncing the read-state with the mbox, if both options are reliably implemented? And for the latter, do you feel the need to be able to move messages from other Thunderbird folders into the local mbox, which I haven't implemented yet or is syncing read-state and individual deletes sanely more than enough already?

No, I have no need to copy/move from other Thunderbird folders into the Unix mbox.

I do not understand the first question. What do you mean by read-state and pruning or syncing it with the mbox? I can imagine the following states of the mbox: "No new mail since thunderbird has last checked it, all mail moved to the global inbox, mbox empty", "New mail has arrived since the last check but it has not yet been moved to the global inbox" (and possibly an additional state like "mbox not yet opened").

Is there any mileage in thinking of "movemail" as protocol in it's own right?
i.e. you kind of treat "movemail" sources (ie external maildirs/mboxes) like you'd treat an external server. Configuration would be a matter of picking the maildir(s)/mbox(es) you wanted to track, and Thunderbird would slurp messages out of them into it's own folders. So there'd be a separate folder subtree in the GUI for movemail, just like how IMAP/Exchange accounts appear.
(To avoid confusion after comments 30 and 31, I'm thinking of movemail as a receive-only protocol here, not for adding messages back to mailspools).

I haven't looked at how the old movemail was implemented, but I'm very wary of it being hacked back in as special-case code.
POP3 is a bit like that at the moment: It's another protocol... but also it isn't. It piggybacks on local folders. It maintains it's own state outside the database. And there's a bunch of special handling to complicate folder implementation and means things aren't as orthogonal as you'd like. It's messy and pops (hehe) up nasty surprises now and then.

Doing movemail as a separate protocol has the benefit of isolating all the synchronisation policy and config and locking strategy from the folder storage code. The recent Exchange work provides a good template.

Of course, if you think of movemail always being something that just watches a single mailspool and feeds messages into an existing TB local folder, and don't think it shuld have it's own folder(s), then maybe thinking of it as a protocol is not the right approach...

Just wanted to throw it out there, and figure out the overall shape before we get bogged down with details.

So there'd be a separate folder subtree in the GUI for movemail, just like how IMAP/Exchange accounts appear.

Exactly what I built, but how to handle migration between different computers isn't perfect yet.

You need to log in before you can comment on or make changes to this bug.