Closed Bug 1789424 Opened 2 years ago Closed 2 years ago

CalDAV Google Calendar hitting rate limits

Categories

(Calendar :: Provider: CalDAV, defect)

Thunderbird 102
defect

Tracking

(Not tracked)

RESOLVED MOVED

People

(Reporter: mbagnara, Unassigned)

References

Details

Attachments

(1 file, 1 obsolete file)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0

Steps to reproduce:

  1. I created a calendar using my Google CalDAV server
  2. I was redirected to the Google Oauth2 provider for signing in
  3. The calendar synchronized
  4. After a couple days the oauth2 session must have expired because now the calendar does not synchronize any longer. It fails to authenticate and I see a 403 exception in the error console. Removing and re-adding the calendar results in the same.

Actual results:

Calendar: CalDAV: Error doing webdav sync: 403 CalDavRequestHandlers.jsm:477
Calendar: There has been an error reading data for calendar: work. However, this error is believed to be minor, so the program will attempt to continue. Error code: DAV_REPORT_ERROR. Description: There has been an error reading data for calendar: https://apidata.googleusercontent.com/caldav/v2/REDACTED/events. It has been disabled until it is safe to use it.

Expected results:

The calendar should request me to authenticate a new session via Oauth2 so that I can obtain a new token.

Component: Untriaged → Provider: CalDAV
Product: Thunderbird → Calendar
Summary: CalDAV Google Calendar 403 → CalDAV Google Calendar HTTP 403 should trigger re-authentication

(In reply to mbagnara from comment #0)

...

Thank you for filing this bug report. I'd like to ask for some more information to help us narrow down the source of this bug:

  1. Do you experience this bug with only one calendar or more than one?
  2. Is this problem consistent? Can you reproduce easily or does it seem to be arbitrary.
  3. How many Google calendars do you have on your profile already? Are they in use with a lot of events?
  4. Is it limited to Google CalDav calendars? Have you experienced this with other providers? Are you able to try?
  5. Are there any repeating events on the offending calendar(s)?
  6. Did the 403 error have any response body? What did it say?
  7. (Edit) Additionally, does your subscription to Google services fall into one of these categories?
    a. Paid subscriptions up to 60 days after meeting the payment threshold
    b. Trial accounts
    c. Users of the legacy free edition of G Suite
    d. Customers of Google for Nonprofits

I'm trying to determine if this is strictly an authentication issue or perhaps a rate limit problem. If it turns out to be the latter, re-authenticating probably won't resolve the issue.

If you feel comfortable getting more technical with Thunderbird, you can enable the calendar.debug.log pref that would provide you with a little more information useful in further determining want went wrong.

I've been experiencing something similar to this, and I've attached an error console log with the calendar debug and verbose prefs set to true. It's been lightly sanitized so the email addresses are not correct.

I am curious about the original poster's answers to comment #1 as well as those will help determine if my issue is the same as this one or not.

Status: UNCONFIRMED → NEW
Ever confirmed: true

What I've just posted is a debugging/test patch for :sancus to try, now we just need to capture the problem again.

Flags: needinfo?(mbagnara)

Hi there

Thank you for filing this bug report. I'd like to ask for some more information to help us narrow down the source of this bug:

Do you experience this bug with only one calendar or more than one? I only have a single remote calendar on Google that  I can test it against, however I  have a separate raw CalDAV server that works fine. 
Is this problem consistent? Can you reproduce easily or does it seem to be arbitrary. It is not consistent. Syncing on cold launch is slow, but occasionally this problem occurs.
How many Google calendars do you have on your profile already? Are they in use with a lot of events? I only have the builtin Thunderbird and the external Google Calendar..
Is it limited to Google CalDav calendars? Have you experienced this with other providers? Are you able to try? I can probably create a Microsoft Office account to test it on Outlook.com
Are there any repeating events on the offending calendar(s)? Yes there are many repeating events each week.
Did the 403 error have any response body? What did it say? I don't have the payload saved but I will try to capture the response payload the next time I see this issue.
(Edit) Additionally, does your subscription to Google services fall into one of these categories? I am currently using a corporate GSuite account through my employer. (A)
Flags: needinfo?(mbagnara)
Attachment #9296406 - Attachment description: WIP: Bug 1789424 - Return early in CalDAV request handlers if the server sends an error code → WIP: Bug 1789424 - Return early in CalDAV request handlers if the server sends an error code.
See Also: → 1792923

Comment on attachment 9296406 [details]
WIP: Bug 1789424 - Return early in CalDAV request handlers if the server sends an error code.

Revision D158228 was moved to bug 1792923. Setting attachment 9296406 [details] to obsolete.

Attachment #9296406 - Attachment is obsolete: true

It's beyond time the bug title was updated to reflect the real problem here, since this seems to be the bug that others are being marked as duplicates of.

In response to the original title, Google is using the 403 response incorrectly (IMO) to indicate that we've hit the rate limit for using their service. It's not actually an authentication problem. We've asked them to raise the limit.

Summary: CalDAV Google Calendar HTTP 403 should trigger re-authentication → CalDAV Google Calendar hitting rate limits

Hi,
I don't know if I can help, but I meet the same problem with 2 different Google Calendars.
Although error 403 in the error console, with the same error messages.

And I meet this message too :
"There has been an error reading data for calendar: work. However, this error is believed to be minor, so the program will attempt to continue. Error code: READ_FAILED"

Hi,

For a few hours now, Google calendars seem to synchronize correctly !

We are still waiting on Google to raise the limit. It may not happen until next week. Problems will be most frequent during peak times for us, which are 7-9am GMT. They should be less frequent at other times.

Shouldn't additionally some kind of throttling be introduced? It makes me nervous to be so dependent on Google's good will. The issue I see is that, even if Google raises the limit as announced, the more people use Thunderbird for calendaring, the higher the risk that the issue appears again.

I have no clue how far we are from the allowed rate, but if all Thunderbird clients would reduce their hit rate when such issues appear, it might not be noticeable to the individual (if my calendar fills in 5 instead of 4 seconds doesn't really matter, but it's a decrease of 20% of the hit rate) but might be sufficient to go overall below the threshold set by Google.

(In reply to Jerome from comment #13)

For a few hours now, Google calendars seem to synchronize correctly !

Yes, it goes in waves, see https://bugzilla.mozilla.org/show_bug.cgi?id=1792048#c11 and it makes sense given the explanation:

  1. more and more people use calendaring and start to synchronize
  2. at some point in time hit rate is above threshold
  3. Thunderbird calendars start to fail and, at some point in time, stop syncing (disabled)
  4. hit rate then goes down below threshold, calendars start again to sync, people re-enable their calendars
  5. go back to point 1

Hello, i think we have exactly same problem. Some of my users are reporting it. It happens mostly with calendars where is more events (hundrets or low thousands), or when users have more calendar connected at once.

Sometimes will help disable and enable calendar again, sometimes Thunderbird must be restarted.

If you will found any solution, at least something to be Thunderbird able recover itself without user's interraction. We will be very happy for that.

The problem was more or less gone for a few days but has appeared again today.

Another idea to (slightly) alleviate the issue: add an option to synchronize individual calendars. People might/will less tend to synchronize all their calendars but only the one(s) they need really, reducing the hit rate and hence the issue at hand.

TB 102.3.1
I have the same issues. Multiple calendars sync with TB but they are available on/off "at random"

TB 102.3.2

S T I L L does not reliably sync with Google Calendar. Adding event to a day that contains events bumps the older event from TB view. Not filterable in the list. Days with events are not bold in the little calendar date picker.

NOT an issue with eM Client calendar.

Additionally, old events moved to an archive calendar are showing up in the future. Events moved from 2018 to 2022 are showing up as 2020 to 2023 and the days are all wrong.

Hi,
Seems to be OK now ?

This is fixed now, the limit has been raised.

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Resolution: FIXED → MOVED

Wouldn't such situation happen again in the future?
Wouldn't the 403 error code received from Google be accompanied by a message like "User Rate Limit Exceeded"?
Shouldn't that allow TB to identify the issue during the sync and handle it directly by temporarily suspending sync accordingly without disabling the calendar completely but notifying end-user via an alert in the UI that the sync is temporarily suspended with the reason? Is it really required for the all Calendar to become disabled from the end-user point of view, especially if Offline Support is enabled? Shouldn't TB retry to sync automatically later without user interaction until fixed by itself?

(In reply to Richard Leger from comment #25)

Wouldn't such situation happen again in the future?

We'll keep an eye on it. The limit has been raised to 50x its prior amount though, so we should be good for some time.

Wouldn't the 403 error code received from Google be accompanied by a message like "User Rate Limit Exceeded"?

Well, part of the confusion here is that Google is misusing errors. 403 has a distinct meaning, it means you are unauthorized to access the resource at the URL. The correct error code for rate limiting is 429. And in addition, they weren't including any additional information in the body of the response, it was just blank. So no, the answer is Google's error messages for CalDav are terrible and you can't expect them to make any sense.

Shouldn't that allow TB to identify the issue during the sync[...]?

Improvements to the way Calendar handles sync errors were made in Bug 1792923 as a result of this bug.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: