Closed Bug 1792078 Opened 3 years ago Closed 2 years ago

Editing item descriptions introduces extraneous newlines on Google Calendar

Categories

(Calendar :: Provider: CalDAV, defect, P1)

Product:
Calendar
Component:
Provider: CalDAV
Version:
Thunderbird 102
Type:
defect

Tracking

(thunderbird113 fixed)

Status:
RESOLVED FIXED
Milestone:
114 Branch
Tracking Flags:
Tracking Status
thunderbird113 --- fixed

People

(Reporter: garret, Assigned: leftmostcat)

References

Details

(Keywords: ux-consistency)

Attachments

(1 file)

Bug 1792078 - avoid extraneous description newlines for Google Calendar. r=#thunderbird-reviewers
48 bytes, text/x-phabricator-request
wsmwk
: approval-comm-beta+
Details | Review

Steps to reproduce:

After moving to the new recent 100.x series (I think it was then), Thunderbird switched to some newfangled rich text editor for calendar events. Every time I save an event and come back, newlines are added all over the place leaving everything a confusing mess.

I type some nice paragraph such as the following (not the exact text):

blah blah foo bar (and more foo bar); this is foo; this is bar
foo and bar
bar and foo

Actual results:

Thunderbird decided to randomly add line breaks, such as:

blah blah foo
bar (and more foo bar); this

is foo; this is bar
foo and

bar
bar and foo

Currently I have no way to predict where Thunderbird is going to throw in line breaks.

Expected results:

I should have got back:

blah blah foo bar (and more foo bar); this is foo; this is bar
foo and bar
bar and foo
(We're not even talking about rich formatting!!)

Why does everyone try to re-implement a word process and break things royally? If you'd just provide plain text and not screw up the lines, that would be 10 times better than this.

Or you could allow us to enter raw Markdown and then show it rendered in the view (using a common library), like Stack Overflow does.

Why oh why are you adding line breaks all over the place every time I edit a calendar event?

It seems to wrap more after colon and semicolon characters; and before a left parenthesis. But not always. And there seems to be no reason for it to have wrapped in the first place.

This is so odd. Why would it want to add newlines?

And if this is storing everything in HTML, then we would expect fewer newlines!

What format is it using on the backend? Surely you didn't go invent some new rich text format did you? Or RTF from 1990-ish?

Just use raw Markdown and let it be. There are enough bugs with the calendar to not have to wrestle with someone deciding to reinvent a word processor.

Hey Garret, I'm seeing this happening as described on a Google CalDAV calendar. 102.3.0 (64-bit), Win10, same on 107.0a1 (2022-09-25) (64-bit).
However, it's not happening on a local Thunderbird calendar. The editor looks the same for both, so it's unlikely (although not impossible) to have different behaviour for CalDAV on the Thunderbird side.
I would suspect that it's Google who breaks this (along https://support.google.com/calendar/thread/29336031/extra-line-breaks-on-event-description?hl=en).

Sancus, do you have someone to look into this?
Garry, can you confirm what type of calendar (local/CalDAV etc.) and from which provider you are using?

STR

  • On a Google CalDAV calendar in TB, create event
  • add event description with some paragraphs of continuous text, e.g. from https://www.lipsum.com/feed/html.
  • Save and close
  • Re-open or re-edit the event.

Actual

  • Extra line breaks (or paragraphs) before the text and between paragraphs. That's odd!
  • continuous text converted to fixed line length (which is not completely random, but still very odd).

Expected

  • no extra line breaks/paragraphs before text and between paragraphs
  • no hard line breaks added in continuous text
Severity: -- → S3
Component: Upstream Synchronization → Provider: CalDAV
Flags: needinfo?(sancus)
Flags: needinfo?(garret)
Keywords: ux-consistency
Product: Thunderbird → Calendar
See Also: → 1659363

Yes, I can confirm that I'm using Google Calendar. But I don't completely understand why that might be an issue.

Can I ask what format you're using on the back-end for rich text? Or do you do like Thunderbird, and store it as plain text until some rich text is used?

If you store things as plain text if there is no rich text formatting, then I can understand why line breaks would show up if a third party (e.g. Google) added them. But if you're rendering some underlying rich text, then I don't understand why line breaks would be rendered inside a semantic unit such as a paragraph — unless something on the backend is actually breaking paragraphs up into multiple blocks!

Flags: needinfo?(garret)

I synced my Google Calender with a third-party service to see what would show up. Most of the older (like a year ago) events had normal, plain text details. But then around August (which I suspect is when Thunderbird got upgraded), I start seeing things like this:

<body>blah; blah blah blah; blah <br>foobar
<div>blah blah; blah blah; blah 
blah blah blah<br>
blah blah and more blah blah</div>
</body>

That is atrocious. It's not even HTML. It's no format at all. It's crap.

Who is putting these nonsense tags in my events? (Is Google doing this? It doesn't make sense that Google would be doing this, especially since it only started happening a few months ago. But it's possible.) I don't yet know what program is the source of this … this mess.

It is looking more and more like this is Thunderbird completely corrupting the events content with this … junk.

Here is the experiment I just did:

  1. Set up Thunderbird and Fastmail Calendar both to sync from Google Calendar.
  2. In Fastmail, add an event with a description that looks somewhat like the following. You'll see that it shows up fine both in Google Calendar online and in Thunderbird Calendar.
Record Locator: XXXXXX; Ticket # XXXXXXXXXXX; $XXX.XX
20XX-XX-XX: Moved to 20XX-XX-XX; Ticket # XXXXXXXXXXX; resulting credit (XXXXXXXXXXX) $XXX.XX.
20XX-XX-XX: Moved to 20XX-XX-XX; Ticket # XXXXXXXXXXX; resulting credit (XXXXXXXXXXX) $XXX
20XX-XX-XX: Moved to 20XX-XX-XX, paying an extra $XX
20XX-XX-XX: Moved to 20XX-XX-XX, Ticket # XXXXXXXXXXXfor $XXX, receiving a credit of $X.XX.
  1. In Fastmail Calendar online, edit the event description to the following. You'll see that it still shows up fine both in Google Calendar online and in Thunderbird Calendar.
Record Locator: XXXXXX; Ticket # XXXXXXXXXXX; $XXX.XX
20XX-XX-XX: Moved to 20XX-XX-XX; Ticket # XXXXXXXXXXX; resulting credit (XXXXXXXXXXX) $XXX.XX.
20XX-XX-XX: Moved to 20XX-XX-XX; Ticket # XXXXXXXXXXX; resulting credit (XXXXXXXXXXX) $XXX

foo

20XX-XX-XX: Moved to 20XX-XX-XX, paying an extra $XX
20XX-XX-XX: Moved to 20XX-XX-XX, Ticket # XXXXXXXXXXXfor $XXX, receiving a credit of $X.XX.
  1. Now go to Thunderbird Calendar, and edit the same event, deleting the "foo" and adding "bar", like this:
Record Locator: XXXXXX; Ticket # XXXXXXXXXXX; $XXX.XX
20XX-XX-XX: Moved to 20XX-XX-XX; Ticket # XXXXXXXXXXX; resulting credit (XXXXXXXXXXX) $XXX.XX.
20XX-XX-XX: Moved to 20XX-XX-XX; Ticket # XXXXXXXXXXX; resulting credit (XXXXXXXXXXX) $XXX
20XX-XX-XX: Moved to 20XX-XX-XX, paying an extra $XX

bar

20XX-XX-XX: Moved to 20XX-XX-XX, Ticket # XXXXXXXXXXXfor $XXX, receiving a credit of $X.XX.
  1. Go back to Fastmail Calendar online, and here is what you'll see:
<body>Record Locator: XXXXXX; Ticket # XXXXXXXXXXX; $XXX.XX<br>20XX-XX-XX:
Moved to 20XX-XX-XX; Ticket # XXXXXXXXXXX; resulting trip credit 
(XXXXXXXXXXX) $XXX.XX.<br>20XX-XX-XX: Moved to 20XX-XX-XX; Ticket # 
XXXXXXXXXXX; resulting trip credit (XXXXXXXXXXX) $XXX<br>20XX-XX-XX:
Moved to 20XX-XX-XX, paying an extra $XX
<div><br>
bar</div>
<div><br>
20XX-XX-XX: Moved to 20XX-XX-XX, Ticket # XXXXXXXXXXX for $XXX, 
receiving a trip credit of $X.XX.</div>
</body>

I am stunned. HTML has been around for about 30 years. We've known not to use <br> for about 20 years. HTML uses the <html> tag, and you certainly don't put a <body> without <html>. Some text is inside <div>, some outside of the <div>. And we should be using <p> tags. I could go on and on. How could a Mozilla product be producing this garbage? (Seriously, objectively speaking, this is garbage. Ask any markup language expert, and they will agree that "garbage" is a valid characterization. It's not just a little error here and there. It is garbage.)

Please, just turn it off. I want to use plain text. How can I disable this? It's corrupting all the calendar entries.

Summary: Thunderbird calendar event rich text editor adds unnecessary line breaks → Thunderbird calendar event rich text editor corrupts descriptions with arbitrary junk tag formatting.

They are even talking about this over on reddit: Buggy: Calendar Event Description Rich-Text Editor:

… In TB 78.x , the big description area was a plain-text editor. It worked flawlessly.
When I upgraded to TB 91.x , it became a rich-text editor. (I'm on MacOS 11.6.x, btw)
… I have to hit enter/return twice to move down each single line. Copy-pasting or deleting lines sometimes moves other nearby text into the wrong place. Unexpected concatenations sometimes occur. Undo/Redo occasionally distort/corrupt the text after a few steps in either direction. It's so frustraing to see my text suddenly flip out when deleting or copy-pasting.
… Does anyone know how I can restore the plain-text editor for the Calendar Event Description area? I'd LOVE that! I looked around in the Preferences, and in the Config Editor (that "about:config" area) but couldn't find anything.

As things are, notes are close to unusable. Until this is fixed, is there any way to have the notes be plain text? I tried using <preformat> but even that was messed up by a close and reopen.

I use google calendar and caldav. If I edit the event in google calendar then I see the 'right' formatting in Thunderbird. If I then edit the event in Thunderbird the formatting gets hosed in both places.

This is even more broken than it seemed; now it is losing/overwriting data.

  1. Connect to Fastmail Calendar.
  2. In Thunderbird create an event on Fastmail Calendar on an arbitrary date (e.g. e.g. January 1) with two lines in the description:

This is really a test.
Then there is more stuff.

  1. In Fastmail Calendar you will see that the formatting is gone. Nevertheless, insert a new "foo" line:

This is really a test.
foo
Then there is more stuff.

  1. In Thunderbird, the mouse flyover for the event correctly shows all three lines with no formatting (as in step 3 above), but if you try to edit the event in Thunderbird it still shows the formatting and the "foo" line is missing (as in step 2 above)!

This is really a test.
Then there is more stuff.

  1. If you then append a new "bar" line, it will erase the "foo" on Fastmail Calendar, which never showed in Thunderbird's edit event dialog (only in the Thunderbird mouse flyover).

This is really a test.
Then there is more stuff.
bar

So when syncing with a CalDAV server that doesn't handle rich text, Thunderbird gets confused, keeps inconsistent data cached, and even loses data (the "foobar" line above) added on the CalDAV remote server!

This is broken, corrupting data, and losing data. Please put this at critical priority.

I see no feedback on this bug report since Sept 26. What does that mean? Is it considered invalid? Not worth fixing? Something else?

From my point of view, this bug makes using notes in events nearly useless. The only way they work "reasonably" is if I edit them in Google calendar and sync them back. I would think that this behavior should provoke a response of some kind.

I see no feedback on this bug report since Sept 26. What does that mean?

There has been no feedback whatsoever from Mozilla on this ticket. Maybe it means they couldn't care less that they completely broke calendar event editing.

There has been no feedback whatsoever from Mozilla on this ticket.

Sorry, that part was incorrect: someone did reply. I had forgotten. But then they just abandoned it.

Thomas D., did you just abandon this ticket? Sancus, why did not not respond to Thomas D.? Does no one care that such a big piece of functionality is completely broken—not to mention being implemented so shoddily to begin with?

Flags: needinfo?(sancus)

The plan is to fix this for 115 by improving the editor and adding a way to switch back and forth between the rich text editor and the plaintext one. So far I haven't seen very many reports of problems with it, but I'm not saying it works great the way it is.

This is not likely going to make it into a 102 minor version. It WILL be in an upcoming beta version that will be available some time before 115, however.

@Andrei Hajdukewycz: Thank you.

(In reply to Andrei Hajdukewycz [:sancus] from comment #13)

The plan is to fix this for 115 by improving the editor and adding a way to switch back and forth between the rich text editor and the plaintext one. So far I haven't seen very many reports of problems with it, but I'm not saying it works great the way it is.

This is not likely going to make it into a 102 minor version. It WILL be in an upcoming beta version that will be available some time before 115, however.

It seems to be a problem due to interaction with google network calendar, therefore I would guess many users have the same experience. Please provide a quick solution even it is not perfect. An event description with plain text should work without any troubles, rich text and html formatting are nice to have, but not essential.

It seems to be a problem imported by Google, maybe solved by them in future. Therefore please concentrate on a stable plain-text-solution.

I am now looking for some workaround into
https://www.gcaltoolkit.com/

A kind of workaround: install add-on to have web-Google-calendar within thunderbird. Set to view the event list, restrict to calendars needed. Then define and edit the description within Google calendar. Use thunderbird only to display the descriptions.

(In reply to Andrei Hajdukewycz [:sancus] from comment #13)

The plan is to fix this for 115 by improving the editor and adding a way to switch back and forth between the rich text editor and the plaintext one. So far I haven't seen very many reports of problems with it, but I'm not saying it works great the way it is.

This is not likely going to make it into a 102 minor version. It WILL be in an upcoming beta version that will be available some time before 115, however.

Good to hear the option to use plain text will be available in 115.

Thunderbird Support Forum does post the issue:
https://support.mozilla.org/en-US/questions/1398700

Often the complaint is not displaying as a problem in Thunderbird, it occurs when users access the google calendar and view event 'notes' via another device eg: phone.
They see the notes and all the html coding at same time.
All complaints found in Google forums (and there are quite a few) seem to blame the various 'mobile phone' apps as not being up to date - eg: Apple's iOS calendar app.
The Google forum seem to take this approach - This is beyond Google's control, so I recommend that you post on Apple's forums to request that they add support for HTML formatting in event descriptions for their iOS and macOS apps.

But there is no means of stripping the formatting from within Thunderbird.

When will we get this new version that allows us to turn off the not-even-real-HTML junk that Thunderbird is forcing into my calendar events?

Look at this existing thread:

https://connect.mozilla.org/t5/ideas/thunderbird-calendar-allow-former-plain-text-editor-in-event/idi-p/14332

Here's a quote:

I actively despise the new editor. Strong words I know but it seems completely unfit for purpose.

And someone else:

Editing notes in Calendar is almost unusable now with the Rich text!

Why is it taking so long to get rid of it when it is corrupts content in core functionality?

(In reply to Garret Wilson from comment #18)

When will we get this new version that allows us to turn off the not-even-real-HTML junk that Thunderbird is forcing into my calendar events?

Look at this existing thread:

https://connect.mozilla.org/t5/ideas/thunderbird-calendar-allow-former-plain-text-editor-in-event/idi-p/14332

Here's a quote:

I actively despise the new editor. Strong words I know but it seems completely unfit for purpose.

And someone else:

Editing notes in Calendar is almost unusable now with the Rich text!

Why is it taking so long to get rid of it when it is corrupts content in core functionality?

This query has already been mentioned.

The plan is to fix this for 115 by improving the editor and adding a way to switch back and forth between the rich text editor and the plaintext one. So far I haven't seen very many reports of problems with it, but I'm not saying it works great the way it is.

This is not likely going to make it into a 102 minor version. It WILL be in an upcoming beta version that will be available some time before 115, however.

Please note, the developers are working on the up coming 115 version and fixing all the various issues that have been and are being uncovered in the beta versions. You will find they are aware of the issues, they read bugs that relate to whatever they are working on, but do not spend precious time chatting feedback in the forum. Just because there is not alot of additional comments by developers does not mean it has been abandoned.
You simply have to wait.
I think there has been plenty of good information posted on this problem with excellent explanations and examples. So thanks for posting the information about this bug.

Agree the "new" editor is really not fit for purpose - any purpose. I stayed away from this for a long time (was using V68 until last week), and now regretting the update. Just junk it and go back to plain text till something better comes along. Happy to help testing

Priority: -- → P1
Depends on: 1827267
Assignee: nobody → leftmostcat
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Summary: Thunderbird calendar event rich text editor corrupts descriptions with arbitrary junk tag formatting. → Editing item descriptions introduces extraneous newlines on Google Calendar

Editing item descriptions introduces extraneous newlines on Google Calendar

The new title is not correct.

This is not just about newlines. The newlines issue is just what appears in your UI. If this was only the case of Thunderbird adding newlines to plane text, this would be oh, so simple. But that's not what's happening.

The bigger problem is behind the scenes. Under the covers Thunderbird decides it wants to make some sort of "rich" markup. It doesn't use Markdown. It doesn't use HTML. What does it use? Some arbitrary mess of tags thrown here and there, not conforming to any specification, and corrupting the plain text that the user entered. That is the true problem.

The newlines is just the first indication that a user sees that something fishy is going on.

The root of the problem is arbitrary junk formatting, not just a few extra newlines thrown in.

… adding newlines to plane text …

edit: "plain text", not "plane text" (wish we had a bug system that allowed editing comments)

The issue here is that Google's implementation of ICS is non-conformant and, in order to support the event descriptions produced by Google Calendar, we need to make some adjustments to what we send them and receive from them. So far, the only visible symptom of this that has been discussed is the extraneous newlines. If there are other visible issues, please make those known. Unnecessary tags in the output is considered a non-issue unless it causes either Thunderbird or Google to handle the description incorrectly.

The issue here is that Google's implementation of ICS is non-conformant …

Non-conformant to what?

It is my understanding that RFC 5545 handles the description as plain text:

https://www.rfc-editor.org/rfc/rfc5545.html#section-3.8.1.5

If Thunderbird sticks some other arbitrary roll-your-own format in there, Thunderbird is being non-compliant. Are you asserting that you're using the RFC 5545 ALTREP facility with a reference to a Content-Type:text/html section with valid HTML? I'm puzzled by how you are saying that Thunderbird is compliant in any manner here. Please explain. What Google is doing is beside the point.

… in order to support the event descriptions produced by Google Calendar …

Google Calendar is only one of many calendar servers we could be using. A user could be using Fastmail Calendar, for example. Why is Thunderbird sticking in non-compliant garbage content just to try to do something in Google Calendar (and not even succeeding at that), and breaking all the content in other calendar servers?

If there are other visible issues, please make those known. Unnecessary tags in the output is considered a non-issue unless it causes either Thunderbird or Google to handle the description incorrectly.

You're assuming that everybody is using Google Calendars. Did Google purchase Mozilla and I missed it?

Fastmail has a very good iCalendar server, which actually follows RFC 5545. They have to try to detect and strip out all this garbage Thunderbird is sending. Sometimes they are successful. Sometimes even in Fastmail these weird tags that Thunderbird sticks in still show up (because it's hard to strip out something that isn't even following any specifications). Here is an excerpt from my conversation with Fastmail support a few months ago:

[Fastmail calendar] Events are stored in iCalendar as defined in RCF 5545 - https://www.rfc-editor.org/rfc/rfc5545.html. There is no support for rich text defined in the standard. Nevertheless, some systems do insert various bits of random HTML into the description (notes). Sometimes, the whole thing is HTML and sometimes it could just be random tags contained throughout. Thus when we display events, we try to detect and handle the common cases (converting to plain text) so you can see a reasonable rendering of everything (instead of having to see random HTML tags in between the content).

iCalendar is a standard in RFC 5545 for almost 15 years. Please comply with the standard and allow Thunderbird to interoperate with standards-compliant iCal servers, instead of sticking garbage in the event descriptions just to try to do something cool in Google Calendar.

If my frustration shines through it's because Thunderbird's calendar used to work just fine! Then someone came along and broke it by trying to add something nobody asked for, in a way that didn't follow the standard, apparently without extensive testing, and … just why?

Please just turn plain text back on, and if you want to experiment with some non-standard Google-specific thing with junk content, put it as an experimental opt-in thing that not only detects if Google Calendar server is being used, but also requires that a user specifically turn on "use experimental junk content to do something cool in Google Calendar".

Just put it back like it was for a decade please! It was working fine.

In almost every case, Thunderbird uses ALTREP when dealing with HTML. It is specifically in Google's case that we don't do this, and it's because of Google's handling of ICS. If you edit a description on Google and introduce rich-text formatting to it, then synchronize that event via CalDAV, the ICS content that Google sends uses bare HTML in the description in non-compliance with the spec. This HTML is also ill-formed, as anywhere the user has introduced a line break while editing on Google's site, Google will send \n instead of <br>. Similarly, Google displays URLs as links, but sends them as bare URLs in the ICS (with no <a> tag).

In order to display events from Google appropriately, we interpret the contents of the DESCRIPTION field specifically on events coming from Google as HTML and likewise send events only to Google with bare HTML in the DESCRIPTION field. The extraneous newlines discussed in this issue are a bug with our attempt to handle the newlines Google sends appropriately. For this issue, I am addressing specifically that bug as it was the one I understood from the discussion.

If there are additional bugs, e.g. Thunderbird failing to correctly use ALTREP as laid out in RFC 5545 with any provider/format other than Google, or the HTML included in our ALTREP data URL causing issues with other providers which attempt to use that ALTREP, please do file those as a separate issue.

Sorry, but the above is just wrong. I type text into the description in Thunderbird then save it. I do nothing in Google Calendar. A few minutes later I open the event and the description has been messed up beyond repair, with seemingly random tags and extraneous newlines. If I do edit a description in Google Calendar then a) Google calendar warns me that it is removing rich text encoding, and b) it is correct in Thunderbird until I change it. I do that frequently so the comments are legible.

As for <br> vs \n: in the thunderbird editor I must frequently hit return multiple times to get what appears to be an empty line. I say "appears to be" because it may or may not survive being saved.

Please please pretty please give us the option of plain text. What we have today is unusable.

Sean thank you for getting into the more technical details. I have switched completely away from Google Calendar trying to get around this issue, but if I'm not mistaken I am still seeing arbitrary tags on other systems. If you could explain a bit further exactly what Thunderbird is doing it would help.

In almost every case, Thunderbird uses ALTREP when dealing with HTML.

So you you saying that if I type "foo\nbar", Thunderbird will create an event containing foo\nbar with a separate ALTREP containing some markup of foo<br/>bar? (Note that in ALTREP my understanding is that you specify an Internet media (MIME) type, and this stuff that Thunderbird is sticking in is not text/html, but I digress.)

So is Thunderbird sending only plain text, or both a plain text version and a "rich-whatever" version, to non-Google servers? If just type plain text with no rich formatting whatsoever, we shouldn't need an ALTREP section in the first place. When I create an email in Thunderbird and don't add any formatting, does Thunderbird put <br/> for all the line endings?? No of course not. It just sends a single plain text part:

…
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
…

So for non-Google servers, does Thunderbird do the same thing for calendar events? That is, if I don't add use rich formatting, does Thunderbird leave off ALTREP altogether and just send plain text?

(In reply to Garret Wilson from comment #28)

So you you saying that if I type "foo\nbar", Thunderbird will create an event containing foo\nbar with a separate ALTREP containing some markup of foo<br/>bar? (Note that in ALTREP my understanding is that you specify an Internet media (MIME) type, and this stuff that Thunderbird is sticking in is not text/html, but I digress.)

If you create an event with the description "foo\nbar" in Thunderbird and it's published via ICS (including CalDAV) anywhere other than Google Calendar, then the current intended behavior is that it would send the following:

DESCRIPTION=ALTREP="data:text/html,foo%3Cbr%3E%0Abar":foo\nbar

This is in compliance with §3.2.1 of RFC 5545: https://icalendar.org/iCalendar-RFC-5545/3-2-1-alternate-text-representation.html; per the spec, "A property specifying this parameter MUST also include a value that reflects the default representation of the text value.", which is the foo\nbar at the end.

So is Thunderbird sending only plain text, or both a plain text version and a "rich-whatever" version, to non-Google servers? If just type plain text with no rich formatting whatsoever, we shouldn't need an ALTREP section in the first place.

At present, yes: we provide ALTREP regardless, as conformant implementations should either skip the contents of the parameter if not supported or display it if they choose to. If the inclusion of conformant ALTREP for plaintext descriptions is causing issues, we'd want a ticket filed for that and we'd address it as a separate body of work.

I appreciate the additional technical details. I will confirm whether arbitrary tags are getting in Fastmail calendar's events.

But still I don't get why you're sending that extra junk to non-Google servers, even if in an ALTREP section, if I don't use any rich formatting?

Like I said, Thunderbird email doesn't work that way. If I just use plain text in the mail editor, then there is no multipart (text/plain and text/html) sent—just a single text/plain part. Why are sticking all this nonsense stuff if we don't even use any rich text editing?

This issue suddenly started occurring with Google Calendar when Thunderbird added rich text. We users didn't do anything different. All my old events (from previous Thunderbird versions) in Google Calendar are just fine. It's only after the date that Thunderbird starting doing its "rich text in descriptions" thing that there were problems on Google Calendar.

DESCRIPTION=foo\nbar

Are you saying that today, in 2023, Google Calendar will somehow mess up this plain text?

This whole rich text issue seems to me to be a red herring. If I don't use rich text, I expect Thunderbird to send only plain text, with no extra pretend-to-be-sort-of-like-HTML junk, either in the main description or in an ALTPREP property, like it did for years and years. Are you saying that Google Calendar would then take the plain text, convert it to HTML, and add arbitrary markup? Please explain exactly what you're saying here because I'm a little incredulous and I don't want to put words in your mouth.

What would happen on Google Calendar if Thunderbird simply sent it plain text like it used to before this new stuff was added?

(In reply to Garret Wilson from comment #30)

But still I don't get why you're sending that extra junk to non-Google servers, even if in an ALTREP section, if I don't use any rich formatting?

The logic that emails use to automatically determine if they can send plaintext only is complex and specific to emails. Copying that behavior for calendar events would be a significant amount of work which we'd want to do only if necessary.

What would happen on Google Calendar if Thunderbird simply sent it plain text like it used to before this new stuff was added?

What this ultimately comes down to is that we do need to support the case where users have added rich text formatting on Google, so we can't elect to only ever send plaintext. We can't easily detect when the user has entered text that can be represented correctly in plaintext, and so we'd prefer not to implement that unless we are correctly using ALTREP and it's still causing problems.

If you can't be respectful then you'll be taking a permanent vacation from Bugzilla. The above conversation is completely unacceptable.

Thunderbird is a free email client, if you don't like it, you're free to stop using it at any time. What you're not free to do is demand that developers do what you want. Read the Bugzilla rules, since you seem completely unfamiliar with them. Particularly "no obligation".

https://bugzilla.mozilla.org/page.cgi?id=etiquette.html

If you can't be respectful then you'll be taking a permanent vacation from Bugzilla.

It was not intended to be disrespectful. I think the use of the word "you" was a bad choice on my part; I merely wanted to address the technical implications of the decision, not attack some person. I'll rephrase them:

Comment 1: I wanted to point out that however much one wants to avoid detecting whether there is plain text in Google Calendar, the fact that third-parties clients can sent plain text means one can't avoid the issue.

Comment 2: I added an edit to fix a typo because the bug system doesn't allow editing, and I thought it would be clever and humorous to format the modification the way Thunderbird now formats calendar events. I can see that not everyone finds it humorous, so never mind.

How can I delete those two comments from this system now that I have rephrased them to clarify I wasn't intending to be abusive? I can't find the delete button.

I don't think you can completely delete comments at all, but you could edit the content.

(In reply to Garret Wilson from comment #32)

  1. Moreover didn't address the problem of what happens if a separate client to Google Calendar were to actually send plain text. For example I can use Fastmail to edit Google Calendar events, and it will only send plain text. (Then the situation will be even worse, as I described above, where you have the same event showing up with different text in different contexts, sort of a "split brain" sort of problem if you're familiar with the CAP theorem and such.) Undoubtedly other clients (perhaps an Android calendar client) will also send plain text to Google. And in this case you're back at the problem of detecting whether plain text or rich text was sent. So you didn't avoid it, you just caused other problems.

The Google description field is "not like the others". They put HTML in there. It's expected. If we treat it as plaintext, then we strip formatting and break <a> links among other things. That's not a good experience, speaking as someone who uses Thunderbird+GCalendar with CalDav every day. Something like 30-50% of all users that have a remote calendar use Google for at least one of them.

If Fastmail is stripping formatting and sending plaintext to Google, that's not something we can fix but it doesn't mean that we should also break Google fields. For non-Google calendars, there should not be any issue with sending both plaintext and altrep with HTML in it.

I don't see any other bugs in the above comments, but if there is one, of course we will want to fix it.

I don't think you can completely delete comments at all, but you could edit the content.

Not to get sidetracked here, but I couldn't find a way to edit comments, which is why I added the separate "edit:" comment in the first place. Oh, well. Live and learn.

If Fastmail is stripping formatting and sending plaintext to Google …

Fastmail isn't "stripping formatting". They don't put formatting in to begin with—they send it as plain text (like the RFC calls for). What I'm trying to get at is that you can have multiple clients accessing the same Google Calendar server at the same time, each adding events. Some non-Thunderbird clients may sent events with plain-text descriptions, and Thunderbird will have to consume them. So I just don't see how Thunderbird consuming plain-text descriptions from other clients is any different than if Thunderbird were to send descriptions in plain text and consume its own events. (And if I just type and add newlines, with no additional formatting, I would have expected Thunderbird to send plain text.) Maybe I'm missing something here. I still think the only real solution is to create some algorithm to detect rich text from the server in this case.

If we treat it as plaintext, then we strip formatting and break <a> links among other things. That's not a good experience …

I hear where you're coming from, and if Google is putting HTML-ish stuff in the description without use of ALTREP, then shame on them. But "force everything to be rich text" is not the approach I would take, and even if it were I would send formatting that was a little more in line with specification best practices, shall we say.

Anyway, thank you Andrei and Sean for helping to explain the technical issues a little more. I've switched from Google Calendar altogether trying to avoid the issue and stick with Thunderbird if I can. If I discover that there is a bug that corrupts content going directly to Fastmail, I'll file a separate bug as you mentioned. In the meantime, if you'd like further feedback on how I would recommend to do things, feel free to ask in this ticket (giving in-depth technical details) and I'll be happy to give my opinion, trying to not to add cheeky humor that can be misinterpreted.

@:leftmostcat:

I looked over the code in WIP: Bug 1792078 - avoid extraneous description newlines for Google Calendar. At the risk of sounding pompous and assuming I follow what the code is doing (there are a lot of external dependencies), it seems your solution is on a good track. Is restoring the timestamp in calViewUtils.jsm near line 390 to avoid a possible infinite loop of cosmetic updates? Will it permit what is stored locally and at Google to diverge?

Thanks for the "Opportunistic fix" for multiple enters required.

Question: when can we expect to see these changes? I would be happy to run a prerelease/beta to test them once such a thing is available.

Is restoring the timestamp in calViewUtils.jsm near line 390 to avoid a possible infinite loop of cosmetic updates? Will it permit what is stored locally and at Google to diverge?

I don't believe there's any possibility of an infinite loop; if we send Google an event with our changes, it will stay that way until it's edited on Google again. There also shouldn't be any possibility of divergence other than the preexisting sync delay.

Thanks for the "Opportunistic fix" for multiple enters required.

Found a potential solution while investigating the rest of the bug and had hit that bug myself, so seemed mete.

Question: when can we expect to see these changes? I would be happy to run a prerelease/beta to test them once such a thing is available.

I'm awaiting some feedback on a bugfix in Firefox code that's necessary for this work and then we'll land this in a beta. I can be sure to post here once that beta is available.

Attachment #9327846 - Attachment description: WIP: Bug 1792078 - avoid extraneous description newlines for Google Calendar. → Bug 1792078 - avoid extraneous description newlines for Google Calendar. r=#thunderbird-reviewers
Keywords: checkin-needed-tb

Pushed by geoff@darktrojan.net:
https://hg.mozilla.org/comm-central/rev/013de2263ecd
avoid extraneous description newlines for Google Calendar. r=mkmelin

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Keywords: checkin-needed-tb
Resolution: --- → FIXED
Target Milestone: --- → 114 Branch
See Also: → 1802791

Comment on attachment 9327846 [details]
Bug 1792078 - avoid extraneous description newlines for Google Calendar. r=#thunderbird-reviewers

[Approval Request Comment]
Regression caused by (bug #): not really a regression per se
User impact if declined: editing of calendar events on Google Calendar will continue to cause odd behavior
Testing completed (on c-c, etc.): manual and automated testing done to verify bugfix
Risk to taking this patch (and alternatives if risky): some risk of unintended behavior with Google Calendar events; could wait until 114b, but more testing is probably better

Attachment #9327846 - Flags: approval-comm-beta?

Comment on attachment 9327846 [details]
Bug 1792078 - avoid extraneous description newlines for Google Calendar. r=#thunderbird-reviewers

[Triage Comment]
Approved for beta

Attachment #9327846 - Flags: approval-comm-beta? → approval-comm-beta+
Duplicate of this bug: 1802791

As of Thunderbird 102.11.0 this still seems to be broken even when other CalDav calendars besides Google Calendar are on the back end. This is exasperating.

Please do the following:

  1. Create a Fastmail account.
  2. Connect your calendar to Thunderbird using CalDav.
  3. Add the following event:
2023-xx-xx: Record Locator: XXXXXX, ticket xxxxxxxxxxxxx, $xxx.xx, paid with flight credit xxxxxxxxxxxxx ($xx.xx), leaving $xx.xx for new trip credit xxxxxxxxxxxxx.
2023-xx-xx: Changed to 2023-xx-xx, ticket number: xxxxxxxxxxxxx.
  1. Open your calendar on Fasmail and edit the event. You'll see something like this:
2023-xx-xx: Record Locator: XXXXXX, ticket xxxxxxxxxxxxx, $xxx.xx, paid with
flight credit xxxxxxxxxxxxx ($xx.xx), leaving $xx.xx for new trip credit
xxxxxxxxxxxxx.
2023-xx-xx: Changed to 2023-xx-xx, ticket number: xxxxxxxxxxxxx.

So it would seem that Thunderbird is still adding line breaks where I don't want them.

Andre, earlier in this ticket you appeared to say that you would offer a way just to turn off the rich text editor altogether. Do you still plan to offer that? Because that's all I want. This rich editor is broken broken broken. I just want to type in text like we could for years and have it work like it did for years. If you read the comments above you'll see that others want this, too.

Why do we have to live with corrupted text? I've done all I can on my end, even switching calendar back ends because of this Thunderbird bug. What is left for me to do as a workaround, other than ditching Thunderbird?

(In reply to Garret Wilson from comment #46)

As of Thunderbird 102.11.0 this still seems to be broken even when other CalDav calendars besides Google Calendar are on the back end.

This patch has not been uplifted to 102. If you want to test it, you can try on beta, but note that there's no path for downgrading profiles, so please back your profile up first.

That said, I understand that you're exasperated, but please also consider that there are people on the other end of the bug report.

That said, I understand that you're exasperated, but please also consider that there are people on the other end of the bug report.

Was my comment anything other than respectful? Was it in any way condescending to anyone? I'm honestly asking, because it sincerely was intended just as a polite but honest update, and I'm confused as to why you are reminding me that "there are people on the other end of the bug report".

Is using the word "exasperating" rude? Is it inappropriate to mention that I've tried everything I can to stick with Thunderbird because I'm a loyal user for decades, even switching away from Google to keep Thunderbird? I'm honestly asking what were you referring to so that my reports can be even more polite in the future.

(In reply to Garret Wilson from comment #46)

Andre, earlier in this ticket you appeared to say that you would offer a way just to turn off the rich text editor altogether. Do you still plan to offer that? Because that's all I want. This rich editor is broken broken broken. I just want to type in text like we could for years and have it work like it did for years. If you read the comments above you'll see that others want this, too.

I also said in that comment "This is not likely going to make it into a 102 minor version", so I don't really understand why you're asking about it being broken in 102? Our current development focus is on 115.0 because that will be released in less than 2 months. I can tell you flat out we're not uplifting this fix to 102. It would require an uplift to Firefox ESR and we're not doing that at this stage of the development cycle. Again, that is in line with what I said before.

As far as turning off the rich text editor, I said that based on an incorrect technical understanding. After Sean looked into it more deeply, he found the real problem and wrote a fix, which is in Daily and Beta now. It will be in 115.0 when that is released in(probably) the second week of July.

If there are some other problems not covered by that solution, then a follow up report is fine, but you'd have to test Beta or Daily first.

I don't really understand why you're asking about it being broken in 102?

I see. I should have been clearer about why I was posting a comment. It was not to nag you; it was 1) to provide additional information in the ticket that I wasn't sure was understood, based upon previous comments here, and 2) clarify whether rich text would be disabled as originally promised, because the comments seemed to be contradictory on this point.

  1. Providing New Information

I had understood from Sean's earlier comment that your team believed that this bug only applied to Google Calendar, and that Thunderbird would only send all these erroneous line breaks if it detected Google Calendar as the calendar server. (Maybe I misunderstood what you were saying.) I thought your team might want to know that Thunderbird is in fact sending this bad data even to non-Google clients, which is why this morning I took the time to report the problem and provide steps to reproduce it.

Would you like me to open a separate ticket for Thunderbird sending erroneous line breaks to non-Google Calendar servers, since your team has updated the title of this ticket to indicate it is only a problem with Google Calendar?

  1. Clarifying Whether An Option Will Be Added to Disable Rich Text

The other motivation was clarifying whether your team will in fact be providing the option to remove this rich text stuff altogether.

Six months ago Andre indicated that we would be getting an option to disable this rich text stuff altogether, so that we could go back to plain text as want. However later comments were ambiguous about whether you still planned to do that. So I politely and respectively asked for clarification.

It is only today (unless I missed a comment) that Andre explicitly notified us that the plans have changed and that you no longer plan on offering us what we have asked for from the beginning: a way to turn off all the rich text in the calendar description editor. It's a good thing I asked for clarification, because this was not clear to me before. Other than that, I can only express (as politely and respectively as I know how) my deep disappointment that even after waiting almost a year we can't get the simple thing some of us asked for, which is "an option to turn off the new thing we don't need or want".

The problem is that you keep repeating over and over that you don't like the rich text editor, but "I don't like this feature, let me turn it off or remove it" is not a bug report.

If the feature is causing problems, then that's a bug report! If we have no other alternative to resolve a serious bug, it's reasonable to remove the feature. In this case, we believe we found an alternative. If you find otherwise testing the fix, then absolutely make a report.

Otherwise there's nothing for us to really do here.

"I don't like this feature, let me turn it off or remove it" is not a bug report

You are right. "Let me turn it off" is not a bug report, but rather a feature request. I have thus filed Bug 1833242, referencing the similar feature Thunderbird email already has to disable HTML format for email composition and asking for an equivalent setting in the calendar event editor.

(In reply to Garret Wilson from comment #52)

"I don't like this feature, let me turn it off or remove it" is not a bug report

You are right. "Let me turn it off" is not a bug report, but rather a feature request. I have thus filed Bug 1833242, referencing the similar feature Thunderbird email already has to disable HTML format for email composition and asking for an equivalent setting in the calendar event editor.

The point you're missing is that the rich text editor is not a problem. Plaintext line endings should work just fine with the rich text editor. If they're not, that's an actual bug, and it's one that should be resolved without removing the rich text editor.

Insisting on removing it is actually making this bug more confusing and harder to resolve, if anything.

See Also: → 1833284
See Also: → 1833242
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: