Closed Bug 2043432 Opened 5 months ago Closed 13 days ago

Have Glean consider 408/429 HTTP response codes to be potentially recoverable

Categories

(Data Platform and Tools :: Glean: SDK, enhancement, P2)

enhancement

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: srose, Assigned: janerik)

References

(Blocks 1 open bug)

Details

Attachments

(1 external file)

Currently Glean considers all 4xx errors to be unrecoverable and immediately deletes the associated pings, but not all 4xx errors are necessarily unrecoverable:

  • 404 Not Found errors could be due to temporary issues with the load balancer (e.g. DS-1843) or their ISP (e.g. bug 1870670).
  • 408 Request Timeout spec says "The client MAY repeat the request without modifications at any later time."
  • 429 Too Many Requests is intended for rate-limiting, and doesn't indicate an unrecoverable error.

In particular, what I'm concerned about is another incident with DNS, ISPs, firewalls, or load balancers causing such response codes to be returned for a significant proportion of clients, which would result in immediate permanent data loss with the current setup. IMO it would be helpful for Glean clients to retry in such cases (up to some reasonable limit) so the threshold for permanent telemetry data loss is on the order of hours/days.

We'll need two things to work on this:

  1. A timeline for when this work should be done
  2. A proposal for what the changes should be.

Sean, can you supply these?

Flags: needinfo?(srose)
Priority: -- → P3
  1. A timeline for when this work should be done

How about by the end of this year at the latest?

Of course, sooner would be better than later, as this remains a data loss risk in the meantime. For example, last month there was a noticeable spike in 408 Request Timeout responses which were apparently due to internet issues in Russia (DENG-11384). And the migration of Ingestion Edge to GCPv2 is now underway, which involves some risk of changes causing unexpected behavior (SVCSE-4142).

  1. A proposal for what the changes should be.

I'd be willing to submit a proposal. However, can you clarify what Glean's existing retry behavior is for errors currently considered to be recoverable? For example, is there particular retry timing logic, like exponential backoff?

Flags: needinfo?(srose)

As discussed in today's Glean working group meeting, I suggest we proceed with changing Glean to treat 408 Request Timeout and 429 Too Many Requests responses as recoverable without a formal proposal, as those seem pretty unambiguous.

While 404 Not Found errors might be transient in certain scenarios like the two I mentioned, I'm unsure what other possible scenarios might cause non-transient 404 errors and risk permanently blocking telemetry submission, so for now I'm retracting my request to change Glean's handling of 404 errors.

And I'm clearing the previously assigned priority to indicate this bug should be re-triaged.

Priority: P3 → --

currently we report 40x errors into a broad category. I wonder if we should split out those 408/429, so we actually know how often that happens in practice?

Assignee: nobody → jrediger
Priority: -- → P2

(via GitHub webhook, authenticated as shared-secret)

Summary: Have Glean consider some 4xx HTTP response codes to be potentially recoverable → Have Glean consider 408/429 HTTP response codes to be potentially recoverable
Status: NEW → RESOLVED
Closed: 13 days ago
Resolution: --- → FIXED
Blocks: 2089635
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: