798932 - Some mail.server.default settings are ignored for new accounts

Status: RESOLVED WONTFIX

(Thunderbird :: Preferences, defect)

Description

[Yarny]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0.1
Build ID: 2012090600

Steps to reproduce:

1. Start Thunderbird with a new profile,
   don't create any accounts
2. Go to about:config, set/verify the following options:
    * mail.server.default.delete_by_age_from_server : false
    * mail.server.default.delete_mail_left_on_server : false
    * mail.server.default.num_days_to_leave_on_server : 30
3. Create new POP3 account.  Any account should do, I used
    * name/email: Yarny <[e-mail]>
    * empyt password, don't remember password
    * pop3.localhost.site:110, STARTTLS, encrypted password
    * smtp.localhost.site:25, STARTTLS, encrypted password



Actual results:

Check about:config:
  * mail.server.server1.delete_by_age_from_server is true
  * mail.server.server1.delete_mail_left_on_server is true
  * mail.server.server1.num_days_to_leave_on_server is 14
The account settings dialog ("Server Settings") confirms this.



Expected results:

I would expect Thunderbird to apply the default settings 
to the new account, as it happens with other settings in
mail.server.default.  The keys above are also listed in
<URL:http://kb.mozillazine.org/Mail_and_news_settings>.

Comment 1

[:aceman]

Did you use the account wizard to create the account?
Have you clicked Manual config?

It seems the code really does set the prefs directly, without checking the default values.
http://mxr.mozilla.org/comm-central/source/mailnews/base/prefs/content/accountcreation/accountConfig.js#110

Are these 3 prefs the only ones having the problem?

Comment 2

[Yarny]

> Did you use the account wizard to create the account?
Yes.  I used "Accounts Settings"/"Account Actions"/"Add Mail Account...".  The dialog box which comes up then is what you mean with "wizard", I guess.
Is there another way?  I would appreciate that.

> Have you clicked Manual config?
Yes (it tries to find a nonexisting server otherwise).  Then I fill in all data and click "Done".

> Are these 3 prefs the only ones having the problem?
Well, I'm also using the following prefs without problems:
 * leave_on_server             : true
 * limit_offline_message_size  : true
 * login_at_startup            : true
 * max_size                    : 16
 * numHdrsToKeep               : 2048
Some of these values might be default values anyway, so even these prefs might be affected.  I don't know if other prefs are affected.

Comment 3

[:aceman]

There is an "advanced config" button you can press instead of "done". But I see it has the same effect, e.g. mail.server.server*.num_days_to_leave_on_server becomes 14.

I can look at this, but need a decision from TB guys what is the correct behaviour here. Maybe the current behaviour is intentional and those prefs are not to be used when using the Account wizard.
Maybe they are used in Seamonkey as they have a different wizard.

Comment 4

[rsx11m]

The defaults are still in place for existing accounts established with Thunderbird 2.0 and earlier. For new accounts, the redesigned account wizard as introduced with bug 422814 sets those explicitly to values which are hard-wired in accountConfig.js (the latter of which may not be a good idea to start with, but that's what was checked in for this bug). This also addressed bug 231541 on possible dataloss scenarios with the old default for hte leaveMessagesOnServer attribute.

SeaMonkey is still using the old account wizard (as present in Thunderbird until 2.0), thus the behavior here is different.

Comment 5

[Yarny]

Thanks, rsx11m, for clarification.

> The defaults are still in place for existing accounts
> established with Thunderbird 2.0 and earlier.
Maybe I can help for the decision if I briefly explain my usecase (if not: sorry, please ignore).

I don't have existing accounts, but new users on my system regularly create two or three new accounts in their Tb profile, which point to /shared/ POP3 accounts.  I have a script which sets some mail.server.default keys before the first start of Tb.  However, at the moment I have to explain my users to make sure they switch off all the "delete from server" checkboxes by hand.  Otherwise Tb starts deleting mails from the server after 14 days.

> the redesigned account wizard as introduced with bug 422814 sets
> those explicitly to values which are hard-wired in accountConfig.js
I think it would be a good idea if the hard-wired values become the default values of the mail.server.default keys, and the wizard then honors these keys.  For an ordinary use who doesn't touch the mail.server.default keys, things wouldn't change at all.

Comment 6

[:aceman]

> Thanks, rsx11m, for clarification.
> > The defaults are still in place for existing accounts
> > established with Thunderbird 2.0 and earlier.
rsx11m, can you expand on this? How does it work in code? I don't understand how that would work. I think all accounts have their own settings prefs already and do not reference the default prefs.

> I think it would be a good idea if the hard-wired values become the default
> values of the mail.server.default keys, and the wizard then honors these
> keys.  For an ordinary use who doesn't touch the mail.server.default keys,
> things wouldn't change at all.

I would agree with this. I think I can code it but need an agreement from TB reviewers.

Comment 7

[Magnus Melin [:mkmelin]]

Accounts created NOW get all the prefs set, but for those pre-tb3 accounts if they didn't change the default there would be no setting for them in the prefs and they would therefore follow the default pref.

I think it specifically doesn't use those preferences not to change defaults for existing users, which could lead to bad situations for them (e.g. data loss, or filled up servers).

Comment 8

[:aceman]

OK, but that still does not mean the use case described can't be supported for new accounts.

Comment 9

[rsx11m]

So, possible solutions are to either (1) add a migration code to complement the set of server-specific prefs for existing accounts, or (2) to define an entirely new set like mail.server.create.* only relevant for accountConfig.js and keep the old default prefs as they are (which also would be transparent to SeaMonkey).

Comment 10

[Yarny]

Aha.  With Magnus' explanation, I see why the default prefs cannot simply be changed.  IMHO, solution (2) by rsx11m sound more compatible for users who rely on mail.server.default filling in non-existing values in the server-specific prefs.  If one of the TB developers with proper authority agrees with solution (2), please consider this bug report a feature request for its implementation.

Comment 11

[Yarny]

Attachment: totally untested patch

Ludovic Hirlimann asked me to propose a patch (by mail).  So this attachment is my interpretation of (2).
Disclaimer: I cannot test it, so it might not compile at all.

Comment 12

[:aceman]

In the patch the prefs will only be picked up when using the wizard. But probably not when creating new accounts from code via accountmanager.createAccount(). I don't know if that is a problem.

Comment 13

[Magnus Melin [:mkmelin]]

Comment on attachment 678833 [details] [diff] [review]
totally untested patch

Review of attachment 678833 [details] [diff] [review]:
-----------------------------------------------------------------

I'm not a mailnews/ reviewer.
(There was a lot of js errors with this patch applied, though i didn't check if it was the cause.)

Comment 14

[rsx11m]

Magnus, while that part is located in mailnews/ it's only used by Thunderbird, at least at this time. A couple of the original patches in bug 422814 were approved by philor, who isn't a MailNews Core peer either, strictly speaking.

Thus, I think that either you or Phil could review this patch, or if indeed a MailNews Core peer is needed, ask Standard8 or Neil for review.

Comment 15

[Ian Neal]

Comment on attachment 678833 [details] [diff] [review]
totally untested patch

>+++ b/mailnews/mailnews.js	Tue Nov 06 15:06:19 2012 +0100
As the accountcreation code is only used by TB, maybe the prefs should be in all-thunderbird.js rather than mailnews.js

Comment 16

[Magnus Melin [:mkmelin]]

Well, the code is in mailnews, so mailnews.js seems consistent to me.
I'll take another look at the patch.

Comment 17

[Yarny]

Is there anything I can do so the patch gets reviewed/improved/merged?
(sorry, I don't want to appear unpatient; just want to avoid a deadlock by everybody waiting for someone else)

Comment 18

[[e-mail]]

Comment on attachment 678833 [details] [diff] [review]
totally untested patch

Sorry. but this is not my area of responsibility.

As IanN says, the prefs probably belong in all-thunderbird.js instead.

Comment 19

[rsx11m]

Comment on attachment 678833 [details] [diff] [review]
totally untested patch

Blake, can you have a look at this given that you've worked on the account creation code?

Comment 20

[Blake Winton (:bwinton) (:☕️)]

Comment on attachment 678833 [details] [diff] [review]
totally untested patch

It looks reasonable to me.  I'm going to ask BenB to do the review, since he wrote or touched a lot of this code.

Thanks,
Blake.

Comment 21

[Ben Bucksch (:BenB)]

(In reply to Yarny from comment #5)
> I don't have existing accounts, but new users on my system regularly create
> two or three new accounts in their Tb profile, which point to /shared/ POP3
> accounts.  I have a script which sets some mail.server.default keys before
> the first start of Tb.  However, at the moment I have to explain my users to
> make sure they switch off all the "delete from server" checkboxes by hand. 

You don't need to tell users to change them by hand. This is what the new autoconfig is for: You can set these values in the config XML that TB fetches from the server.
https://developer.mozilla.org/en-US/docs/Thunderbird/Autoconfiguration#Small_company
https://developer.mozilla.org/en-US/docs/Thunderbird/Autoconfiguration/FileFormat/HowTo
https://wiki.mozilla.org/Thunderbird:Autoconfiguration:ConfigFileFormat

The latter shows how you can set these values that you desire, for the accounts that you share.

BTW: Why would you share POP3 accounts? Use IMAP shared folders, please. If you can set up scripts to modify Thunderbird configs, you can also set up an IMAP server - I do that for personal use, too.

Comment 22

[Ben Bucksch (:BenB)]

Comment on attachment 678833 [details] [diff] [review]
totally untested patch

While I understand the rationale, I think this is misdirected. You should set these on an per-domain basis, not globally. We already have the infrastructure for that in place, and it's easier to set up (server-based) than modifying Thunderbird default prefs on each machine.

If you set this globally, you might affect other accounts, e.g. personal accounts or work accounts that were set up under different circumstances. But if you still really really want to do that, I can tell you a trick how to achieve that with a server-side scripts.

Comment 23

[Magnus Melin [:mkmelin]]

The xml config doesn't have an option for setting delete_by_age_from_server to false. But i guess you can use a really large day count (say 50 years or something) to achieve the same thing.

Comment 24

[Yarny]

> This is what the new autoconfig is for
I'll have a look into this.  I need to add, however, that I have no control over the server, so the autoconfig would probably have to be stored locally (served with something like lighttpd?) and Tb must accept that the autoconfig comes from a machine which is different from the actual POP3 server.

> Use IMAP shared folders, please.
You're right, that would be the proper solution.  I don't have a machine for that, sadly.

> If you set this globally, you might affect other accounts, e.g. personal
> accounts or work accounts that were set up under different circumstances.
I'm not sure if I understand that correctly.  I thought that the new settings (mail.server.create.*) do not affect existing accounts.  In fact, even new accounts should behave like before unless someone changes those mail.server.create.* keys.

Comment 25

[Ben Bucksch (:BenB)]

> > Use IMAP shared folders, please.
> You're right, that would be the proper solution.  I don't have a machine for that, sadly.

FWIW, gandi.net offers a domain and email hosting for 2 Eur/month for all your users.

POP3 is not made to run with the "never delete", your access will get ever slower (and eventually overflow, too).

bwinton: WONTFIX? Given that there are better alternatives?

Comment 26

[Blake Winton (:bwinton) (:☕️)]

I'm going to wait to hear back from Yarny, as to whether that solution is satisfactory…

Comment 27

[Yarny]

>> This is what the new autoconfig is for
> I'll have a look into this.
I tested this by putting some test.domain.xml files into my local Tb's isp directory.  From what I understand, this is the only way I can feed these files into Tb without modifying the mail server.  I observed that it cannot influence delete_mail_left_on_server and, as Magnus Melin observed, delete_by_age_from_server.  Based on this, I still will have to instruct my users to change these settings by hand to avoid accidential data loss.  What botters me more is that I have to hardcode a lot of server data into these xml files, e.g., during my testing, Tb refused to honor the xml file if it didn't contain an "outgoingServer" section.  At the moment I leave such settings to my users.  If they misconfigure something, it just won't work and they will find out themselves.  If, however, they misconfigure the settings which I mentioned in my original posting, they might delete mail from the server without ever realizing it.  Finally, I have root access to most, but not all of my users machines, so sometimes I just can't access the isp directory.

Summing up, I guess that the autoconfig is a comfortable way to provide server configuration to the client, but my problem is a "client-side" problem as it does not define the way the client "accesses" its mailbox, but the way it "uses" its mailbox.

> POP3 is not made to run with the "never delete", your access
> will get ever slower (and eventually overflow, too).
The email traffic is low and once a year, someone creates a Tb profile where (on purpose) delete_mail_left_on_server is set to true for all accounts, then deletes old mails.

> I'm going to wait to hear back from Yarny,
> as to whether that solution is satisfactory...
Thanks a lot for your explanations and support.  I'd still be grateful if you could apply my patch, or implement another way to define default values for those config keys.

Comment 28

[Neville Hillyer]

I think Yarny should be congratulated on his excellent work. I was trying something similar when I found this bug. I am now using a slightly simpler version which consists of only replacing the following 9 lines:

      checkInterval : 10,
      loginAtStartup : true,
      // POP3 only:
      // Not yet implemented. { Boolean }
      useGlobalInbox : false,
      leaveMessagesOnServer : true,
      daysToLeaveMessagesOnServer : 14,
      deleteByAgeFromServer : true,
      // When user hits delete, delete from local store and from server
      deleteOnServerWhenLocalDelete : true,
      downloadOnBiff : true,

with:

      checkInterval : Services.prefs.getIntPref("mail.server.default.check_time"),
      loginAtStartup : Services.prefs.getBoolPref("mail.server.default.login_at_startup"),
      // POP3 only:
      // Not yet implemented. { Boolean }
      useGlobalInbox : false,
      leaveMessagesOnServer : Services.prefs.getBoolPref("mail.server.default.leave_on_server"),
      daysToLeaveMessagesOnServer : Services.prefs.getIntPref("mail.server.default.num_days_to_leave_on_server"),
      deleteByAgeFromServer : Services.prefs.getBoolPref("mail.server.default.delete_by_age_from_server"),
      // When user hits delete, delete from local store and from server
      deleteOnServerWhenLocalDelete : Services.prefs.getBoolPref("mail.server.default.delete_mail_left_on_server"),
      downloadOnBiff : Services.prefs.getBoolPref("mail.server.default.download_on_biff"),

I also do a few edits in defaults/pref.

This appears to work OK for clean installs (without history preferences) and for adding accounts in Account Settings.

I am not too concerned about using preferences intended for the default server as the defaults for new accounts but I wonder if my simpler approach could result in difficulties if account settings are edited later.

Any comments would be appreciated.

For those concerned about keeping mail on POP servers I can say that I see no appreciable lag with about 1,500 messages on Gandi (4 months - 122 days). However, a few years ago I was aware of delays with 40,000 messages on a local UNIX server.

Neville

Comment 29

[Yarny]

Is there still a chance to get the patch merged?
(Sorry if I appear to be unpatient again.)

Comment 30

[Blake Winton (:bwinton) (:☕️)]

Well, I'm not going to mark the bug as a wontfix, but you're going to have to convince BenB to give you an r+ before it can land.  (Or, what's more likely, work with BenB to figure out an acceptable solution to your problem that he also approves.  I'm not sure what form that solution will take, but the system is a little flexible, so I suspect other solutions that you can both agree on exist.)

Comment 31

[:aceman]

Comment on attachment 678833 [details] [diff] [review]
totally untested patch

Review of attachment 678833 [details] [diff] [review]:
-----------------------------------------------------------------

I like the idea (but haven't tested the patch yet).

::: mailnews/base/prefs/content/accountcreation/accountConfig.js
@@ +108,2 @@
>        // POP3 only:
>        // Not yet implemented. { Boolean }

Any idea what this comment means? Are the settings bellow this actually used?

Comment 32

[Yarny]

Hi Ben,

given Blake's Comment #30, do you have any suggestions to improve my patch so it becomes acceptable for inclusion in Thunderbird?  As I outlined already in Comment #27, the auto-config mechanism is unfortunatelly not a feasible solution for my problem.

Thanks -- Yarny

Comment 33

[:aceman]

I imagine this functionality could also be used in a test profile, where the developer often creates test accounts of one type (e.g. POP3). Those account may reference nonexistent servers (the developer tests e.g. account manager behaviour, not server connections). In that case he may want to always have the account pick up some predefined settings (from prefs.js, not hard-coding the values in accountConfig.js as that would get into his patch via hg qref).

Comment 34

[xyz123]

This is a bug which causes data loss but it seems developers don't care and it seems it wont be fixed.

So as a standart user is there anything I can do to fix this?

I really didn't understand the insistence on not fixing this.

Instead of fixing a bug that cause data loss, developers say "use imap why you use pop3" then why you don't remove pop3 functionality if you won't fix its bugs and tell people "use imap instead of pop3" ?

Comment 35

[xyz123]

If it does nothing then why you put these options to about:config?

Comment 36

[Magnus Melin [:mkmelin]]

Because they are needed for accounts created predating the "new" (tb3+) account setup.

Comment 37

[Ben Bucksch (:BenB)]

It's not dataloss, because it does exactly what POP3 is supposed to do: Fetch the email, then save it locally. That's how POP3 works, that's how it's designed, and that's how it works in Thunderbird. If you want to keep you mail store only on the server, use IMAP. IMAP was invented, exactly because POP3 is not suited for that, and there are significant operational disadvantages in using it like that. If you want to abuse it like that, do not expect support.

Comment 38

[xyz123]

You definitely don't know what you're talking about. It's dataloss because it causes data loss. And what abuse you're talking about? Problem is Thunderbird doesn't respect user settings that it offers, not how POP3 or IMAP works. If these settings are there and if they don't have any effect then why they are there?

If a program gives us an option to select A or B and if it always works with A in spite of we selected B then there's a bug that should be fixed. You can't say "If there's a bug then don't use it" which exactly what you're saying all the time in this bug report.

And it's not your business to dictate users what they should use. As I said this has nothing with POP3 or IMAP working ways. This a bug with Thunderbird user settings.

Anyway do whatever you want. Yeah you're right everyone else is wrong. You're the only people in the world who can think what is right and everyone should follow your words and live with your decisions.

Have fun with your playground. I won't write anything else because it seems it wont be any effect.

What a pity!

(In reply to Ben Bucksch (:BenB) from comment #37)
> It's not dataloss, because it does exactly what POP3 is supposed to do:
> Fetch the email, then save it locally. That's how POP3 works, that's how
> it's designed, and that's how it works in Thunderbird. If you want to keep
> you mail store only on the server, use IMAP. IMAP was invented, exactly
> because POP3 is not suited for that, and there are significant operational
> disadvantages in using it like that. If you want to abuse it like that, do
> not expect support.

Comment 39

[:aceman]

xyz123, the options (prefs) do work correctly, just not the way you/Yarny want them to work. For your intended working (just asssuming, as you haven't provided a description of what you actually want, just came up with a dataloss suddenly), Yarny has prepared a patch. Maybe if you described your usecase we could better understand why the functionality is needed to you.

Comment 40

[Yarny]

Hi Ben, good to see you're still watching this bug.

> If you want to abuse it like that, do not expect support.
I wouldn't consider this abuse.  RFC 1939 says "normally, mail is downloaded and then deleted", but it allows for other workflows and even explicitely considers them in Section 8.  Comments 28 shows that there are more users who cannot replace POP3 with IMAP for some reason.  Neville Hillyer is even using a patched version of Thunderbird to work around this problem (I'm not tech-savvy enough to do so).

Thunderbird already supports this workflow now, my patch is merely about allowing the user to activate this workflow by default for new accounts (it does not change the default).  In view of this, would you please clarify why my patch is not suitable for inclusion in Thunderbird.  If there is a problem with the code, I would like to fix it as soon as possible, if you solely reject the feature of user-controlled default values for POP3 handling: please reconsider.  While you're not needing it, there are others who do, and this feature shouldn't harm anyone.

Thanks

Comment 41

[Ben Bucksch (:BenB)]

Yarny,

Nobody is preventing you from using POP3 like this.
* You can configure it manually
* You can write a script for your organization to configure account prefs automatically.
  (that's what many admins do)
* autoconfig has support for that
  * https://developer.mozilla.org/en-US/docs/Thunderbird/Autoconfiguration#Small_company
    Just use autoconfig, it's a merely matter of placing a single static file on your
    www server (not mail server).
  * You can also place the files in your TB folder, as you've already tried in comment 27.
    If it didn't work, then probably because there was something wrong with your file.
    (I think you can even skip the outgoing server and use the existing server,
    not sure about that, check the source.)

So, that's 4 options, for a real edge case. I think that's ought to be enough. Sorry, but I don't want to add another way, for a real fringe case.

I've explained above in comment 22 what's wrong with the patch in general.

FWIW, the patch isn't good either, we shouldn't access prefs in the object init(), but rather do that elsewhere, and patches must be tested before being submitted for review, but I don't think we should take *any* patch for this, so that's why I didn't dive into that.

I *would* like to commend you for making the effort of coming up with the patch. I honor that.

(I really advise you to buy IMAP service for 5 bucks/month instead of fiddling around like this.)

Comment 42

[Yarny]

Thanks for your clear words Ben.

Given Blake Winton's comment 30 there seems to be no way this bug report will eventually lead to a change in Tb, and it should probably be closed as WONTFIX now (I'd do this but I'm not sure if I'd overstep my authority).

Comment 43

[Neville Hillyer]

(In reply to Yarny from comment #42)

What a pity Yarny my sympathies are with you in this matter.

Thanks Ben for the interesting and very useful resource at:
https://developer.mozilla.org/en-US/docs/Thunderbird/Autoconfiguration 

It is unfortunate that users are being coerced into using POP when IMAP may be more suitable for their needs.

It may not be appropriate for me to say much more here as my main Macs are G4+ with Leopard and hence, apart from my Ubuntu box, I am forced to use very old versions of Thunderbird. For various reasons I now use TenFourBird 10.0.11 from:
http://sourceforge.jp/projects/tenfourbird/releases/?package_id=12932

Comment 44

[Yarny]

Closing as suggested in comment 42 after a year of silence.