1059988 - Multiple semicolon-separated addresses no longer work in to/cc/bcc fields, empty/null fields should be ignored (deprecated/invalid syntax Bug 242693)

Status: RESOLVED FIXED

(Thunderbird :: Message Compose Window, defect)

Description

[Eric Jain]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0 (Beta/Release)
Build ID: 20140716183446

Steps to reproduce:

Prior to release 31, I was able to paste multiple comma- or semicolon-separated addresses into a to/cc/bcc field and Thunderbird was smart enough to split them up. Now, Thunderbird appears to mangle the addresses.

Comment 1

[Thomas D. (:thomas8)]

Eric, I understand the concern. A few more details and more accurate description would have been helpful, we really have enough work... :)

Anyway, this is what happens:

- comma-separated addresses, after pasting + Enter, get split up correctly as they should, both with quoted and unquoted display names, even quoted display names with comma
"Hans, Meier" <[e-mail]>, Peter Müller <[e-mail]>, Susi Sau <[e-mail]>

Hans, Meier <[e-mail]> (double quotes visually removed in composition window for simplicity, but kept/added if required when saving draft and sending)
Peter Müller <[e-mail]>
Susi Sau <[e-mail]>

- in TB31, semicolon-separated addresses remain as they are, no splitting up

As a matter of fact, semicolon-separated addresses used like comma-separated addresses are INVALID syntax, mostly because semicolon's are actually allowed syntax for another feature, RFC822 groups. They are named lists with semicolon in the physical mail header:

listname: recipient1 <[e-mail]>, recipient2 <[e-mail]>;
listname: recipient1 <[e-mail]>, recipient2 <[e-mail]>;, Normal Address <[e-mail]>
The following is also using the groups syntax with an empty list:
undisclosed-recipients:;

Although apparently we can't handle that group syntax yet, Bug 919953. Your request to support semicolon as an alternative separator (albeit invalid) was implemented in Bug 242693; looks like this regressed for TB31. Other relevant bugs:  bug 83521, bug 110605. bug 94676.

- TB24: semicolon-separated addresses are split up in recipient pane upon pressing ENTER

As a workaround, paste in your favourite text editor / word processor first, replace semicolons with commas (all in one go, search/replace all), then re-copy, re-paste.

Comment 2

[Thomas D. (:thomas8)]

We should test how we handle illegaly semicolon-separated addresses in mailto-links, which was enabled by bug 258653.

Comment 3

[Eric Jain]

My bad: When pasting a list of comma-separated email addresses, the fields scrolled down, so it looked like all but the last address had been discarded; hadn't noticed the scroll bar way to the right. Tried semicolons after that, which of course didn't work.

Comment 4

[Thomas D. (:thomas8)]

Well actually, I think Bug 242693 enabled semicolon-separated lists as a courtesy to users, so it should work even though it's deprecated/invalid syntax.

Comment 5

[Thomas D. (:thomas8)]

Probably regressed by js mime header works done by Joshua; I think I even saw a comment for semicolons issue somewhere.

Comment 7

[Steuard Jensen]

Can someone give me a little more information on the work that led to this regression? (A bug reference, maybe?) How straightforward would this be to fix, following the example of bug 242693? (Not that I'm promising anything yet...)

Context: My organization just switched to Outlook as its primary email client, so lists of addresses generated internally are now displayed with semicolons as separators. Doing a search/replace every single time I want to paste those into Thunderbird is maddening. For all of the reasons described in bug 242693, I think it's essential for Thunderbird to either (ideally) provide compatibility with this non-standard syntax or (if necessary) fail gracefully rather than sending obvious gibberish to the mail server.

Comment 12

[Wayne Mery (:wsmwk)]

I could imagine that enterprise users in particular would not be happy about this

Comment 13

[Anje]

Using Windows Vista and Thunderbird version 45
Confirm this bug has reoccurred.

method of manually inputting email address using a semi-colon, with and without a following space fails because it is not seen as two emails, but as one badly formed. You can see this as it has inserted speach quotes on sending.
eg:
[e-mail]; [e-mail]
becomes: "[e-mail]; me"@foobar.com
Tested by adding a name and chevrons either side of actual email address using semi-colon and also got a failure of the same type.
eg:
Anje <[e-mail]>; Anje <me@[e-mail]>
becomes
"Anje <[e-mail]>; Anje" <me@[e-mail]>

Woraround:
Confirm that using a comma instead of semi-colon works as expected creating two email addresses correctly.

Comment 14

[Ralf-Peter Rohbeck]

I was just bitten majorly by this. A reply from a customer with several people on the distribution had semicolons separating the names and a Reply-All just wouldn't go through. Servers in the chain kept rejecting the mail (for good reasons.) I could finally send the reply by teasing the email addresses apart by hand.

When I tried by hand I got this reply within minutes but this didn't happen with the initial failure:

Delivery has failed to these recipients or groups:

"[e-mail]; me"@gmail.com
The recipient's e-mail address isn't correct. Please check the e-mail address and try to resend the message. If the problem continues, contact your helpdesk.

Comment 17

[toni]

Hi dears, 
sorry for coming back. I noticed that for somehow reason Gmail accounts can't be added to Thunderbird.
Therefore it's compulsory to download and install the last version.
I noticed that the feature about the row separation has not been yet re-added. This is the reason I am still on 24.6.0 version.
Now Gmail will force me to upgrade to the version 31 or laterhttps://support.mozilla.org/en-US/kb/thunderbird-and-gmail

Using several times multiple, recipient I need the feature working back again like v. 24.

Can you please help me ??

Comment 18

[toni]

The issue is still there.
I can't understand how I am the only desperate who suffer it
This is the list of emails for testing.
Address are on different rows without comma

[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]

Once inserted in TB v. 24.6.0
task regularly performed 

To: [e-mail], [e-mail], 
 [e-mail], [e-mail], [e-mail], 
 [e-mail], [e-mail], [e-mail]

Once inserted the same on v.52.6.0

An error occurred while sending mail. The mail server responded:  
5.1.2 The recipient address <, [e-mail]> is not a valid RFC-5321
5.1.2 address. f206sm3068151wmf.26 - gsmtp.
 Please check the message recipient "", parks"@pipponzo.com" and try again

and the TOTAL MESS:

"To: info"@pipponzo.com, [e-mail],
 ", sysadmin"@pipponzo.com, [e-mail], [e-mail],
 ", parks"@pipponzo.com, [e-mail], [e-mail]



It seems a real DEEP bug and not an additional feature as somebody continue stating !
Possible I am the only one experiencing this issue ?
Possible that anybody else has this issue ?

Comment 19

[Magnus Melin [:mkmelin]]

Eh, do not include the "To: " in the paste

Comment 20

[toni]

obviously not

Comment 21

[Thomas D. (:thomas8)]

(In reply to toni from comment #18)
> The issue is still there.
> I can't understand how I am the only desperate who suffer it
> This is the list of emails for testing.
> Address are on different rows without comma
> 
> [e-mail]
> [e-mail]

Toni, I tested your scenario and it works correctly for me.

STR
1) copy list of email addresses separated by line breaks (CRLF on windows)
2) paste into TB's To-field
3) press Enter

Actual result
2) upon pasting, TB converts line breaks into a comma-separated list, single line, i.e. all recipients now in a single To-field
3) After pressing Enter (with cursor on the comma-separated, single-line list of recipients), recipients are correctly spread with 1 recipient per line, commas removed, and nothing wrong seen.

This does not work for you? Meaning there must be something else involved, addons, display names with comma, quotation marks, etc.

Comment 22

[Thomas D. (:thomas8)]

(In reply to Thomas D. (currently busy elsewhere) from comment #21)
 Actual result
> 2) upon pasting, TB converts line breaks into a comma-separated list, single
> line, i.e. all recipients now in a single To-field

Aceman, I wonder how many recipients can fit into a single input field, and if we're casting any error message if not all of them fit - can you test this with your 4000 recipients list? STR see comment 21.

Comment 23

[Nathan Neulinger]

Space/newline separated emails work for me in current 59, including ones with ,'s in quoted name portion "LN,First" <[e-mail]>. Semicolon separated (such as copied from windows/outlook/etc.) do not.

Comment 24

[Anje]

Using TB v 52.6.0
Performed same test as advised and completely agree with Thomas D.
Even if I did not press 'Enter' and left all in one line, all sent ok.



However.... I notice you refer to a mess eg: ", parks"@pipponzo.com,

I can replicate this, but only if I have two commas between email addresses.

This time I'm copy pasting from the 'view source' of email.
I selected one of the emails I sent which has a comma after each name.
I clicked on 'More' and select 'View source'.
I noted that the last email address of four was on another line, just like the text you copy pasted.It looked perfectly ok, there was no second comma, but....
If I copied all four of those email addresses in one go from the source view 'TO' email addresses and paste into a new Write TO field:
Noticed an additional comma had now been inserted between the last one on upper line and first one on new line.
So when sent it is identical to the mess you are talking about.


If the list of emails in a MSWord document happens to have a comma after the email address eg: ending .com,
then after copy pasting, it will also produce two commas and send up with same result.

Could Toni please test this: I would like you to do the usual copy paste into a TO field.
Then before doing anything, carefully read through those inserted emails. 
Do you see any that have two commas between some email addresses?

Comment 25

[toni]

Dears, 
this file once copy and pasted in To creates issue of the comma before the name.
I hope it  helps to replicate the issue I have.
The file has been made with MS Word 2010 

Download "Pipponzo_Email_List_TEST_2.doc" at #4shared -"  
https://www.4shared.com/office/jA0gDNQ9ca/Pipponzo_Emai_List_TEST_2.html

I select ALL data from the *.doc file and I paste into To field of TB v. 52.6.0, as it is. I do not paste without formatting or other . I do everything the same on  v. 24.6.0 and v. 52.6.0. First wrok always, second always error reported  below

This is the error I got all the time I repeat the action as above. this doesn't happen on TB v. 24.6.0 all later versions yes.


Si è verificato un errore durante l'invio della posta. Il server di posta ha risposto: 
5.1.2 The recipient address <,[e-mail]> is not a valid RFC-5321
5.1.2 address. q29sm3784969edd.25 - gsmtp.
 Controllare il destinatario del messaggio "",tombaura"@pipponzo.com" e riprovare

... can't understand why !

Comment 26

[Anje]

re :To creates issue of the comma before the name.
Yes it is supposed to create a comma, that is not the issue, I believe the blank lines are creating additional commas.

The image is poor at link (I'm not attempting a download as you can get anything except what you want), but I can see that there are two email addresses then a space/clear line, followed by a group of three email addresses then another break followed by another three etc.

I'm thinking that the email addresses which are going wrong are always those which have a line or paragraph break above them. So an extra comma is being inserted for the blank line.

After pasting in the email addresses into the TO field, you need to check each one to see if there are any email addresses with two commas, like this example:
To: [e-mail], [e-mail],,[e-mail], [e-mail], [e-mail],,[e-mail], [e-mail], [e-mail]

This would correspond with the email addresses you mention as shown in example.

In the Word document, remove those blank lines between email addresses in the list.
The list needs to be a list with an email address on each line, not a broken list with various gaps and spaces between email addresses.
Then try the copy paste method.

Comment 27

[Thomas D. (:thomas8)]

(In reply to Anje from comment #24)
Thanks a lot Anje for testing this out and reporting back your findings, much appreciated.

(In reply to toni from comment #25)
Hi toni, thanks for sharing your user experience and sorry for the fact that your use cases were failing after updating TB and the inconvenience caused by that. We did re-code the handling of email addresses completely somewhere along the line, so that's what probably caused the change in behavior. Assuming that your report of the old behavior is correct, I must say that you've been quite lucky that copying email addresses mixed with empty lines and Word comments straight into TB just worked... but so it did, and, I guess it still should, but alas - atm it doesn't. As for Word comments, fortunately they get morphed into trailing spaces which then get trimmed by TB.

Here's what causes the problem:
-> blank lines between email addresses in the copied text from Word

Here's how to avoid that problem:
If you can just remove your empty lines between email addresses before copying, everything will be fine. For easy removal:
- press Ctrl+H
- Find what: ^p^p^p, Replace with: ^p,
- Replace All
(you can type ^p by pressing ^, then space, then p; select email address section first if there's other content in your doc which you want to preserve). Then another round:
- Find ^p^p, Replace with: ^p
- Replace All
Now your empty lines are gone, and you can safely copy and paste your list of email addresses (separated by line breaks/ paragraph marks) into TB, and it's going to work just fine if there are no commas in addition to the line breaks.

Comment 28

[Thomas D. (:thomas8)]

That said (comment 27: avoid empty lines between a list of LF-separated email addresses pasted into TB's recipient field), TB should really handle this case more gracefully.
In fact, imo there are three address parsing bugs here:

1) when pasting a list of email addresses separated by line feeds into To-field, empty lines (LF LF or whatever they may be) inside the list must not be converted to double comma |,,|. Let's just trim them away please.

2)a) when pressing Enter so addresses get normalized into one address per recipient field, two or more commas |,,|, even when found in to-field, and not in quotation marks or angle brackets, must not result in addresses having a comma in front of them |,[e-mail]|; extra commas must be trimmed away
2)b) - I'd also think that we could probably trim leading commas in the user part of an email address |,[e-mail]| away, rather than converting them into |",user"@pipponzo.com|;
- also trailing commas at the end of the input field |[e-mail],| should be trimmed;
- but we may want to preserve leading commas in display name: |,Thomas <[e-mail]>|.
I'm not totally sure about 2)b), e.g. edge cases like |thomas,,@asdf.com|, which we may want to convert using "".

3) when pasting a list of email addresses separated by comma AND line feeds, they should not should not be converted to double comma |,,|. We must never create two commas ourselves.

Note the second part of this bug's current summary: "...empty/null fields should be ignored".

Jörg, how hard would it be to fix some of this? I'm thinking how you helpfully fixed some of the search algorithms...

Comment 29

[Thomas D. (:thomas8)]

(In reply to Thomas D. (currently busy elsewhere) from comment #28)

> 2)a) when pressing Enter so addresses get normalized into one address per
> recipient field, two or more commas |,,|, even when found in to-field, and
> not in quotation marks or angle brackets, must not result in addresses
> having a comma in front of them |,[e-mail]|; extra commas must be
> trimmed away
> 2)b) - I'd also think that we could probably trim leading commas in the user
> part of an email address |,[e-mail]| away, rather than converting
> them into |",user"@pipponzo.com|;
> - also trailing commas at the end of the input field |[e-mail],|
> should be trimmed;
> - but we may want to preserve leading commas in display name: |,Thomas
> <[e-mail]>|.
> I'm not totally sure about 2)b), e.g. edge cases like |thomas,,@asdf.com|,
> which we may want to convert using "".

Rephrasing what we could do for 2):
Any occurence of two or more commas, unless inside quotation marks |"thomas,,"@asdf.com| or inside angle brackets |<thomas,,@asdf.com>|, should be trimmed down to a single comma when pressing Enter to normalize into single recipient lines, or when sending.
Single comma is a separator, but two or more subsequent commas without quotation marks are invalid syntax anyway; and I think we can expect user to put quotation marks himself for those rare edge cases where two or more subsequent commas are really wanted, and it's not helpful trying to quote this ourselves, as we're more likely to get that wrong than to get it right.
|thomas,,[e-mail]| should become
thomas
[e-mail]

Comment 30

[Thomas D. (:thomas8)]

(In reply to Thomas D. (currently busy elsewhere) from comment #29)
> (In reply to Thomas D. (currently busy elsewhere) from comment #28)
> 
> > 2)a) when pressing Enter so addresses get normalized into one address per
> > recipient field, two or more commas |,,|, even when found in to-field, and
> > not in quotation marks or angle brackets, must not result in addresses
> > having a comma in front of them |,[e-mail]|; extra commas must be
> > trimmed away
> > 2)b) - I'd also think that we could probably trim leading commas in the user
> > part of an email address |,[e-mail]| away, rather than converting
> > them into |",user"@pipponzo.com|;
> > - also trailing commas at the end of the input field |[e-mail],|
> > should be trimmed;
> > - but we may want to preserve leading commas in display name: |,Thomas
> > <[e-mail]>|.
> > I'm not totally sure about 2)b), e.g. edge cases like |thomas,,@asdf.com|,
> > which we may want to convert using "".
> 
> Rephrasing what we could do for 2):
> Any occurence of two or more commas, unless inside quotation marks
> |"thomas,,"@asdf.com| or inside angle brackets |<thomas,,@asdf.com>|, should
> be trimmed down to a single comma when pressing Enter to normalize into
> single recipient lines, or when sending.

...and leading or trailing single comma should also be trimmed away, unless in quotation marks or angle brackets.
Something like that.

Comment 31

[Thomas D. (:thomas8)]

4) I'm also surprised that |[e-mail][e-mail]|, where _ is space, gets converted into [e-mail]_,[e-mail] when pasting, any need to preserve leading/trailing spaces which aren't inside quotes or angle brackets?

Comment 32

[toni]

Hi dear,
OI appreciate you started debating actively on what I still consider an ISSUE and not a welcomed feature (differently from many senior programmers here present) introduced in the passage between v. 24.6.0 and the next ones. 

Is it outrageous to ask to copy and paste that code line which allowed TB to understand empty lines as just a comma-comma space like => , , or ,, 

This didn't affected the multiple recipients use, even with space in the between, maybe done to seprate different groups of recipient - isn't it ? -and recover it back to his proper functionality before and included version v. 24.6.0 ?
Do you think you can do ? 
If you perform this major change you will adjust and annoying and nasty issue which couple perfectly with the one of the spell checking unsolved problem!

I really hope you can fulfil my request, for the rest of the world, and I will be here available to test the corrected version.
I wait for your positive answer.
Don't bury again this ISSUE in the forgotten issues limbo.
Even becuse NOW GMAIl refused to work with new accounts on version early than 31 and, on the same side, Yahoo accounts refuses to work with v. 52 and later.

This is messy.

Do you think it's time to give up TB ? 

Is it true rumour about TB project is going to be abandoned ?

Comment 33

[toni]

Sorry my answer is directed to anybody involved here but it starts from Ange answer [e-mail]

Comment 34

[toni]

Sorry it was Thomas D. answer ... No delete comment  button available

Comment 35

[Thomas D. (:thomas8)]

(In reply to toni from comment #32)
> Hi dear,
> OI appreciate you started debating actively on what I still consider an
> ISSUE and not a welcomed feature (differently from many senior programmers
> here present) introduced in the passage between v. 24.6.0 and the next ones. 

No offense meant, Toni, and I think you didn't get me correctly.
I explicitly agreed with you in my comment 28 that I think TB's current behaviour isn't practical and can be considered a bug, or issue as you call it (wrong behaviour of the program).
That said, it's also true that even though it was working before, we never promised that irregular empty lines and double commas which you paste into a single-recipient field would work, but I agree with you it would be nice if it did (again).

> Is it outrageous to ask to copy and paste that code line which allowed TB to
> understand empty lines as just a comma-comma space like => , , or ,, 

Unfortunately, it's not that simple. The old parser was probably written in C++, now it's JS, and it has been completely rewritten so we can't just copy the old line into the new code. It needs someone to find the right spot between 7 million lines of code and come up with a patch. Finding the right spot is much harder than writing up those few lines of code to fix it. But it's definitely doable, that's why I have tried to nail down the problem into something actionable and asked Jörg to look into it.

> This didn't affected the multiple recipients use, even with space in the
> between, maybe done to seprate different groups of recipient - isn't it ?
> -and recover it back to his proper functionality before and included version
> v. 24.6.0 ?
> Do you think you can do ? 
> If you perform this major change you will adjust and annoying and nasty
> issue which couple perfectly with the one of the spell checking unsolved
> problem!

There are thousands of bugs on record, and in the big scheme of things, having to remove empty lines between your recipients before copying them into TB sounds quite minor, compared e.g. to things like dataloss and search failures. Is there a bug for the spell check problem you're seeing? If yes, pls mention the bug # (but no discussion here), if not, pls file one.

> I really hope you can fulfil my request, for the rest of the world, and I
> will be here available to test the corrected version.
> I wait for your positive answer.
> Don't bury again this ISSUE in the forgotten issues limbo.

I've just tried to get some traction on your issue (which is only indirectly related to the actual issue of this bug which starts out from semicolons problem).

> Even becuse NOW GMAIl refused to work with new accounts on version early
> than 31 and,

Do you really expect old versions of TB to handle the latest changes from email providers forever? That's not realistic.

> on the same side, Yahoo accounts refuses to work with v. 52 and later.

If there's no bug for that yet, please file one.

> This is messy. Do you think it's time to give up TB ?

If your only concerns about TB are this bug and one spelling bug, that doesn't sound all that messy...
We're trying our best to maintain TB and keep it stable and secure, and we're always fixing a lot of bugs in every release. Unfortunately, paid manpower is still extremely limited, so we are mostly volunteers, but less than a handful of regular code contributors, up against a large legacy of bugs.

> Is it true rumour about TB project is going to be abandoned ?

No. After Mozilla all but dropped their funding some years back and essentially withdrew paid staff, we've created a new income stream from donations and TB council is in the process of hiring, which includes adding to the developing manpower. Related information e.g. here: https://blog.mozilla.org/thunderbird/

Comment 36

[Jorg K (CEST = GMT+2)]

I'm not familiar with this.

I pasted |[e-mail], [e-mail], [e-mail]| and |huhu <[e-mail]>, hihi <[e-mail]>, hoho <[e-mail]>| and got three separate lines.
Repeating with |[e-mail]; [e-mail]; [e-mail]| it doesn't work.

Processing is most likely here:
https://dxr.mozilla.org/comm-central/rev/a8eecfe6de793af00e98cb1488515199c5fb73fb/mail/components/compose/content/addressingWidgetOverlay.js#298

Comment 37

[Jorg K (CEST = GMT+2)]

Hmm, looks like what I said was wrong. I changed that to |var addressArray = inputValue.split(/[,;]/);| and it made no difference. Further reading shows that the splitting is done in SplitRecipients():
https://dxr.mozilla.org/comm-central/rev/a8eecfe6de793af00e98cb1488515199c5fb73fb/mailnews/compose/src/nsMsgCompFields.cpp#603
and EncodedHeader() changed with the introduction of JSMime. So that makes it harder to fix.

Interesting reading here:
https://dxr.mozilla.org/comm-central/rev/a8eecfe6de793af00e98cb1488515199c5fb73fb/calendar/base/modules/calUtils.jsm#246
Quote: splitRecipients(..) behaves wrongly ...
So our Calendar friends knew it all along and are doing their own splitting :-(

Alfred, care to dig through JS Mime to see how easy that can be fixed?

Comment 38

[Anje]

Info for Toni:
Bugzilla people will now have a look at this issue.
In the meantime, to avoid the issue you are experiencing, I suggest you access the Word document with all the email addresses and remove all of those empty lines between email addresses. So you have one email address per line with no blank lines between. Doing this should help to resolve the issue you are experiencing allowing you to use an up to date version of Thunderbird.

It does not mean the bug is resolved, but it should help you to use Thunderbird and not experience those issues in the meantime.

Comment 39

[toni]

Dear Anjem
thank for your reply.
Meanwhile I continue using v. 24.6.0 without the possibility to add new Gmail accounts in the old version and with the other tremendous issue with Yahoo account on the v. 52 and later.
Would you open another bug pst  about that ? 
Thanks

Comment 40

[Thomas D. (:thomas8)]

(In reply to toni from comment #39)
> Dear Anjem
> thank for your reply.
> Meanwhile I continue using v. 24.6.0 without the possibility to add new
> Gmail accounts in the old version 

Yes Toni, you've said that before and it won't change anything to repeat it. You can't expect old software versions to keep up with current changes of email providers forever. Btw it's probably the same oAuth problem which you're facing for Yahoo, and we've fixed that for gmail for the new version which you aren't using. Sorry that yahoo is broken as you claim, which is a truly bad thing which should be fixed, but it doesn't change the fact that our manpower is limited, mostly volunteer driven, and sometimes with thousands of other important bugs it happens that important bugs don't get done. 

> and with the other tremendous issue with
> Yahoo account on the v. 52 and later.

Probably bug 1293958, maybe bug 614826 or bug 1337445, see full list:
https://bugzilla.mozilla.org/buglist.cgi?quicksearch=%3Athun%2Cmailn%20yahoo&list_id=14069326

Comment 41

[toni]

I am really mortified in hearing all that ?
Do you think TB has other parallel projects with major investment on the side ? 
TB should remai a major project funded in such a way by uses ...

I am really sad in finding that all my issues have not been addressed yet by any programmes (since 6-7 yrs) and the are not going to improve but ON  THE CONTRARY to get worst.

Dear Thomas here it seems top be the clay pot among iron pots ... Do you understand this rhetorical figure ?
Ho can I help and support ?

the issue that fit are the following
Bug 1337445 and Bug 1293958

Here I am talking about PO3 connection, never IMAP on Thunderbird client to avoid to mess around all email on the cellphone and on the client.

Thanks

Comment 42

[Anje]

Toni
Have you fixed your Word document so it does not have blank unused lines?
If yes, then this will resolve the issue you have as defined in this bug.
Then update Thunderbird so it is compatible with gmail issue.
Yahoo recently needed people to use 'less secure app'option have you done this? If no then try doing so.

Comment 43

[toni]

(In reply to Anje from comment #42)
> Toni
> Have you fixed your Word document so it does not have blank unused lines?
> If yes, then this will resolve the issue you have as defined in this bug.
> Then update Thunderbird so it is compatible with gmail issue.
> Yahoo recently needed people to use 'less secure app'option have you done
> this? If no then try doing so.

Dear Anje, 
I don't want arguing with you. What you proposed is just a workaround. 
Thomas D. understood the issue quite clearly I suppose. You don't, for somehow reasons I don't want to investigate. 

The address list Word input file has to be used as it is. No space removal or any further modification. 
it has been working on TB v. 24.6.0 and it should work on v. 52 or later (v. 60 in development ).
Till that day the issue is clearly a BUG and NOT AT ALL a feature.

This is my definitive point of view.
See about the other bugs related to Yahoo.
I don't find correct to address other parts about the TB issue and BUGS.
Everybody should do his own porper job.
Do you ?

Regards
T.

Comment 44

[Thomas D. (:thomas8)]

(In reply to toni from comment #43)
> Dear Anje, 
> I don't want arguing with you. What you proposed is just a workaround. 
> Thomas D. understood the issue quite clearly I suppose. You don't, for
> somehow reasons I don't want to investigate. 
> ...
> Everybody should do his own porper job.
> Do you ?

We've sorted this out in private mail.
Thanks Anje for trying to help and offering correct workarounds, the ball is now in Toni's court to try them.

Comment 45

[Thomas D. (:thomas8)]

Wayne, could you kindly try to find out with Tony (or link us with another support person) as to why he can't get his Yahoo account to work with the latest version of TB? There's a couple of TB bugs around Yahoo, but Anje suggested to try Yahoo's alleged setting "Allow less secure apps to access Yahoo" as a workaround. Unfortunately, I'm not much into that and I'm suspecting it needs some step by step assistance to try that out or see where the problem really is. See my comment 40. Thanks.

Comment 46

[Thomas D. (:thomas8)]

Sorry for bugspam, accidentally cancelled ni?aceman

(In reply to Thomas D. (currently busy elsewhere) from comment #22)
> (In reply to Thomas D. (currently busy elsewhere) from comment #21)
>  Actual result
> > 2) upon pasting, TB converts line breaks into a comma-separated list, single
> > line, i.e. all recipients now in a single To-field
> 
> Aceman, I wonder how many recipients can fit into a single input field, and
> if we're casting any error message if not all of them fit - can you test
> this with your 4000 recipients list? STR see comment 21.

Comment 47

[Wayne Mery (:wsmwk)]

Matt and Gene have more experience with yahoo - see comment 45

Comment 48

[Matt]

(In reply to Thomas D. (currently busy elsewhere) from comment #45)
> Wayne, could you kindly try to find out with Tony (or link us with another
> support person) as to why he can't get his Yahoo account to work with the
> latest version of TB? There's a couple of TB bugs around Yahoo, but Anje
> suggested to try Yahoo's alleged setting "Allow less secure apps to access
> Yahoo" as a workaround. Unfortunately, I'm not much into that and I'm
> suspecting it needs some step by step assistance to try that out or see
> where the problem really is. See my comment 40. Thanks.

What is this about an "alleged" setting?  It can be located at the bottom of this page https://login.yahoo.com/account/security?.scrumb  Sign into your yahoo web account as necessary.

This appears to have degenerated into a support topic.  I suggest the poster actually uses the support site. https://support.mozilla.org/en-US/questions/new  I do not intend to offer ongoing support in Bugzilla.

If there is a bug here it is lost in the back and forth. I am assuming it is that Thunderbirds should separate addresses separated by a semi colon.  I would postulate that the correct behaviour would be to separate the list based on the list separator defined in the operating system instead of hard coding something unless the RFC actually defines a list separator.

Yep looked at rfc6068 The RFC for the MailTo: specification defines the separator to be used as a comma.  As that is why we accept a list of addresses and parse it,  there is no reason to accept a semi colon defined list.  Even if Microsoft do insist on semicolons not commas for Outlook.

I pasted the list in https://bugzilla.mozilla.org/show_bug.cgi?id=1059988#c18 into a Thunderbird compose window and pressed enter.  The list parses correctly.  So I see no issue with the lists put up by Toni.

My feeling is there is no bug here, even if it is a regression.  The refusal of the user to sanitize their data is not a bug.



https://tools.ietf.org/html/rfc6068

Comment 49

[toni]

(In reply to Thomas D. (currently busy elsewhere) from comment #46)
> Sorry for bugspam, accidentally cancelled ni?aceman
> 
> (In reply to Thomas D. (currently busy elsewhere) from comment #22)
> > (In reply to Thomas D. (currently busy elsewhere) from comment #21)
> >  Actual result
> > > 2) upon pasting, TB converts line breaks into a comma-separated list, single
> > > line, i.e. all recipients now in a single To-field
> > 
> > Aceman, I wonder how many recipients can fit into a single input field, and
> > if we're casting any error message if not all of them fit - can you test
> > this with your 4000 recipients list? STR see comment 21.

Dear Thomas,
I apologise for bothering you.
Did the bug make any progress or as usual nobody has the will to solve it ?
The issue is always the same: someone would define this issue as a feature and not as a bug. I don't know who has the authority to decide BUT I ma pretty sure it can't be considered a feature !

I wait for your reply and I hope you can reactivate the issue solving.

Ciao

Comment 50

[Sylvain]

(In reply to Matt from comment #48)

> If there is a bug here it is lost in the back and forth. I am assuming it is
> that Thunderbirds should separate addresses separated by a semi colon.  I
> would postulate that the correct behaviour would be to separate the list
> based on the list separator defined in the operating system instead of hard
> coding something unless the RFC actually defines a list separator.

It is not really that TB should separate addresses with semi-colons and semi-colons only; it is that it should ALLOW this. Can semi-colons be used inside an email address? Not that I know of. Are millions of people used to using semi-colons because of Outlook? Yes. Then what is the big problem with supporting semi-colons? If Outlook used a period to separate email addresses, and the period was not an allowed character in email addresses, then I would be asking you to support it as an email list separator. The change in the TB source is extremely minor I am sure, and could be reverted easily anytime, so why not? There is no real difference between a comma and a semi-colon in list separation - i.e. they both "feel" the same, just like in text sentences, they separate things. It is hard for me to see a reason NOT to support both.

> Yep looked at rfc6068 The RFC for the MailTo: specification defines the
> separator to be used as a comma.  As that is why we accept a list of
> addresses and parse it,  there is no reason to accept a semi colon defined
> list.  Even if Microsoft do insist on semicolons not commas for Outlook.

Nothing prevents you from going beyond the RFC.

Best Regards,

Comment 51

[Matt]

(In reply to Booze from comment #50)

> > Yep looked at rfc6068 The RFC for the MailTo: specification defines the
> > separator to be used as a comma.  As that is why we accept a list of
> > addresses and parse it,  there is no reason to accept a semi colon defined
> > list.  Even if Microsoft do insist on semicolons not commas for Outlook.
> 
> Nothing prevents you from going beyond the RFC.
> 
Nothing prevents Microsoft from adhering to it. 

I still maintain if we are having a list separator, beyond the comma defined in the RFC, we should look to the operating system to define it.  

If you want to change your list separator to a semi colon,  that is your choice.  You will have what you want.

Comment 52

[Sylvain]

(In reply to Matt from comment #51)
> (In reply to Booze from comment #50)
> 
> > > Yep looked at rfc6068 The RFC for the MailTo: specification defines the
> > > separator to be used as a comma.  As that is why we accept a list of
> > > addresses and parse it,  there is no reason to accept a semi colon defined
> > > list.  Even if Microsoft do insist on semicolons not commas for Outlook.
> > 
> > Nothing prevents you from going beyond the RFC.
> > 
> Nothing prevents Microsoft from adhering to it. 

Yeah, I know, but it's too late now, everyone's used to the semi-colon (sure, rare businesses don't use Outlook, but so many of us are stuck using it every day.) I do not know why they went with the semi-colon at all. At least in my neck of the woods, Microsoft themselves say that the list-separator is the comma.

> I still maintain if we are having a list separator, beyond the comma defined
> in the RFC, we should look to the operating system to define it.  
> If you want to change your list separator to a semi colon,  that is your
> choice.  You will have what you want.

I'd rather not do that, or other things will break, like DOS commands, or Excel (can you imagine, Excel suddenly needing a semi-colon between values in CSV files?) You can always support all three: Commas, semi-colons and the OS list-separator. You can also add a preference for this, something like "support.additional.list.separators" and leave it false by default.

I don't want to be a pain, asking for TB to work "my" way, but so many of us use that darn semi-colon 30 times a day at work. TB is much better than Outlook, but it's hard to remember to use commas in TB - I use email address lists way more at work than at home, so I tend to remember that, and wonder regularly why it's not working in TB.

Best Regards,

Comment 53

[toni]

(In reply to toni from comment #25)
> Dears, 
> this file once copy and pasted in To creates issue of the comma before the
> name.
> I hope it  helps to replicate the issue I have.
> The file has been made with MS Word 2010 
> 
> Download "Pipponzo_Email_List_TEST_2.doc" at #4shared -"  
> https://www.4shared.com/office/jA0gDNQ9ca/Pipponzo_Emai_List_TEST_2.html
> 
> I select ALL data from the *.doc file and I paste into To field of TB v.
> 52.6.0, as it is. I do not paste without formatting or other . I do
> everything the same on  v. 24.6.0 and v. 52.6.0. First wrok always, second
> always error reported  below
> 
> This is the error I got all the time I repeat the action as above. this
> doesn't happen on TB v. 24.6.0 all later versions yes.
> 
> 
> Si è verificato un errore durante l'invio della posta. Il server di posta ha
> risposto: 
> 5.1.2 The recipient address <,[e-mail]> is not a valid RFC-5321
> 5.1.2 address. q29sm3784969edd.25 - gsmtp.
>  Controllare il destinatario del messaggio "",tombaura"@pipponzo.com" e
> riprovare
> 
> ... can't understand why !

Being honest the focal problem is not on semi-colon or comma but why TB new versions, after 24.6, interpret a list of address separated in different lines instead of comma separated addresses in a mess that prevent you to send any email to multiple address TB is not able to sort out from.
I mean why it was working before and later from that version not ?
This mean worsening software capability version by version. It sounds like a useless destructive approach I can nor understand neither support !

Comment 54

[toni]

Anyway no idea and no improvement at all !
It's a mystery !
Deeply sad about that

Comment 55

[Thomas D. (:thomas8)]

(In reply to toni from comment #53)

> This mean worsening software capability version by version. It sounds like a
> useless destructive approach I can nor understand neither support !

I've responded to this via PM:

https://bugzilla.mozilla.org/page.cgi?id=etiquette.html

Semicolon remains a part of this bug's focus, notwithstanding that per my comment 1, we have to balance between allowing semicolon as a list separator with a very different meaning and as address separator (outlook misfeature). Toni's problem (can't copy/paste recipient lists with blank lines) is a minor loss of function at best, with relatively easy workaround (remove blank lines before copying). But as already explained in my comment 28, TB should handle this better: several unfortunate parsing patterns which can be considered bugs. Fixing this isn't trivial unless someone points out the right spot. Jörg's comment 37 has made a starting point.

Comment 56

[Thomas D. (:thomas8)]

(In reply to Booze from comment #50)
> It is not really that TB should separate addresses with semi-colons and
> semi-colons only; it is that it should ALLOW this.

Agreed, imo we should definitely try to allow semi-colon separated addresses so as to avoid needless inconveniences to users with outlook exposure.

> Can semi-colons be used inside an email address? Not that I know of.

Caveat! As explained in my comment 1, semicolons can be used with another distinct function in address headers: RFC822 groups. TB currently isn't very good at handling those, but we must ensure that we don't break that legitimate syntax when allowing semicolons as an alternative address separator. I guess it can be done with some effort.

Basic RFC822 groups syntax:
> listname: recipient1 <[e-mail]>, recipient2 <[e-mail]>;

RFC822 groups syntax combined with comma separated addresses:
> listname: recipient1 <[e-mail]>, recipient2 <[e-mail]>;, Normal Address <[e-mail]>

RFC822 groups syntax (with empty list) used by TB for representing BCC:
> undisclosed-recipients:;

What this means is that we can't just interpret every top-level semicolon as an address separator, as that would break the groups syntax. So the algorithm needs to identify groups first and keep them intact, and only deal with any non-grouping semicolons.

Comment 57

[Sylvain]

(In reply to Thomas D. (currently busy elsewhere) from comment #56)

Good day.

> Agreed, imo we should definitely try to allow semi-colon separated addresses
> so as to avoid needless inconveniences to users with outlook exposure.

I have no idea why Microsoft chose that character, but I at least, am now used to it...

> Caveat! As explained in my comment 1, semicolons can be used with another
> distinct function in address headers: RFC822 groups. TB currently isn't very
> good at handling those, but we must ensure that we don't break that
> legitimate syntax when allowing semicolons as an alternative address
> separator. I guess it can be done with some effort.

Ah sorry, I jumped deep in this thread and missed your comments. Funny you should mention groups, as I subscribe to another bug that is about them actually - a friend wants them fixed, and I thought I'd keep an eye on the bug. I don't use groups myself (all mailing lists I use in TB have a normal list email address, not a group name.) I've never had to use groups in TB, I'm not sure when I would used them (I read RFC822 6.2.6. MULTIPLE MAILBOXES but am no closer to seeing where this is a requirement.) Perhaps Outlook/Exchange communicate to each other using groups when one sends an email to an Exchange mailing list? I've never really looked into it...

> Basic RFC822 groups syntax:
> > listname: recipient1 <[e-mail]>, recipient2 <[e-mail]>;
> RFC822 groups syntax combined with comma separated addresses:
> > listname: recipient1 <[e-mail]>, recipient2 <[e-mail]>;, Normal Address <[e-mail]>
> RFC822 groups syntax (with empty list) used by TB for representing BCC:
> > undisclosed-recipients:;

Well, you could treat all things between semi-colons as groups - ie, split it first by semi-colon, then determine if each part is a group or just an email address - but I'm sure it is more complicated than that (I see ";," separating a group from a regular email address, what a pain.)

> What this means is that we can't just interpret every top-level semicolon as
> an address separator, as that would break the groups syntax. So the
> algorithm needs to identify groups first and keep them intact, and only deal
> with any non-grouping semicolons.

Yeah, things in software are always way more complicated than when we first casually look at them. I should remember that more often.

Thanks for educating me :-)
Best Regards,

Comment 59

[Magnus Melin [:mkmelin]]

Attachment: bug1059988_addr_semicolon.patch

Handling semicolons, unless the input string also has preceding colon, which would be needed for a group. Not going to fix a case of possibly mixing groups and invalid syntax.

Comment 60

[Jorg K (CEST = GMT+2)]

Wow, such an easy win after such a long discussion. Looking good, I'll test it later.

Comment 61

[Jorg K (CEST = GMT+2)]

Magnus, I'm a bit confused by the commit message: handle semicolons as mailbox separators gracefully when pasting into composition.

You can just type it, right? It's just that many users paste like in bug 1336785. There we landed with: account for multiple commas when parsing email addresses.

So here we should do: handle semicolons as mailbox separators gracefully when when parsing email addresses.

You agree?

Comment 62

[Jorg K (CEST = GMT+2)]

Comment on attachment 9090981 [details] [diff] [review]
bug1059988_addr_semicolon.patch

I should have reviewed bug 1336785 better. All addresses generated by splitting at comma or semi-colon are shown in black in the widget as if they were in the address book.

This change by itself is useful but incomplete. Would you like to address it here or maybe better in a new bug. Well, I guess splitting at a comma always worked, just not multiples, the the colour was always wrong.

As I said, I'd do the commit message without the word "pasting" since when testing, I pasted nothing.

Comment 63

[Magnus Melin [:mkmelin]]

Feel free free to adjust the commit message. It may not be technically limited to pasting, but that is probably where the problem arises. You basically never write many addresses on the same line when entering them manually.

I'll leave the colours for another bug. That would be somewhere different code wise.
Sent off to try now: https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=c0df27bd977b76f4fe8fb0b8dcd2659a0da4745c

Comment 64

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/c42051e85e3a
handle semicolons as mailbox separators gracefully when when parsing email addresses. r=jorgk

Comment 65

[Jorg K (CEST = GMT+2)]

Magnus, the side-effect of this change and the one in bug 1336785 is that display names are also altered, so if I enter "a ,, b ; c" <a@b.c> I get "a , b , c" <a@b.c>. Should we change the regexp in the substitution?

Comment 66

[Magnus Melin [:mkmelin]]

Patch in bug bug 1336785 to adjust a bit.

Comment 67

[Jorg K (CEST = GMT+2)]

But you're still changing semicolons to commas in the display string, right?

Comment 68

[Jorg K (CEST = GMT+2)]

Needs further consideration.

Comment 69

[Magnus Melin [:mkmelin]]

Attachment: bug1059988_addr_semi.followup.patch

Only replace when not within a quote.
Adapted from https://stackoverflow.com/questions/15741243/replace-string-everywhere-except-if-its-within-quotes

Comment 70

[Jorg K (CEST = GMT+2)]

Comment on attachment 9091370 [details] [diff] [review]
bug1059988_addr_semi.followup.patch

In all honesty, I don't understand what it does, but it seems to work. I think we really need to comment this. How about:
Explanation from https://stackoverflow.com/questions/15741243/replace-string-everywhere-except-if-its-within-quotes:
This regex is using a positive lookahead that is basically matching 0 or more occurrences of a pair of some text until a double quote is found i.e. ([^"]*"){2} on the right hand side (RHS) of every match of ; .

Which in simple term means replace a ; only if it is outside double quotes since all the matches inside double quotes will have odd number of [^"]*" matches on the RHS.

Or can you write something up that's clearer.

Comment 71

[Magnus Melin [:mkmelin]]

I'd just put that in the commit message. It's good not to make readers overloaded with too long explanations.

Comment 72

[Jorg K (CEST = GMT+2)]

But who will see this in the commit message? Can we come up with some hint in the code? I tried running it through https://regex101.com/ and even that doesn't help.

Comment 73

[Magnus Melin [:mkmelin]]

Who is going to analyze the regexp in the first place? :)

Comment 74

[Jorg K (CEST = GMT+2)]

Comment on attachment 9090981 [details] [diff] [review]
bug1059988_addr_semicolon.patch

Let's take it now together with the other patch.

Comment 75

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/c5590ab6d0c6
follow-up to only replace semicolons with commas in case the semicolons aren't within a quote. r=jorgk

Comment 76

[Jorg K (CEST = GMT+2)]

TB 70 beta 1:
https://hg.mozilla.org/releases/comm-beta/rev/344b5b374a1ddaaa9a5bdf66b37b7c9ace7f0e35 (merged patch)

Comment 77

[Jorg K (CEST = GMT+2)]

TB 68.1.1 ESR:
https://hg.mozilla.org/releases/comm-esr68/rev/1a2c18f909ecfcf391bbd80bc1a11869b0110e1d

Comment 78

[Alexander Bergmann]

I can confirm the problem is solved in Thunderbird 68.1.1.

Comment 79

[toni]

On TB 60.9.0, the officiale tested release now rolling out the issue is still presetn ! How to solve it ?

Comment 80

[Jorg K (CEST = GMT+2)]

Switch to TB 68. We're not fixing any bugs in TB 60.x any more. TB 60.9.0 was definitely the last release in that series.