1765406 - Glean Android not respecting the 500 event limit and ping size limit

Status: NEW

(Data Platform and Tools :: Glean: SDK, defect, P4)

Description

[Travis Long [:travis_]]

Referencing Bug 1762953, ingestion failed for a handful of pings for Focus Android because they were too large (> 8MB). It also appears that they have many, many more events per ping than we allow. :wdh noted a ping with 49968 events... 100 times over what should have triggered a ping.

The scope of this bug covers investigation into why our limits aren't being applied and resolving this so that they are.

As a side effect of this, we may also want to validate that this works as intended on other platforms, as well.

Comment 1

[Jan-Erik Rediger [:janerik]]

:whd, any chance you could provide a ping sample from bug 1762953?
I don't seem to have access to payload_bytes_error.structured so can't load it myself.
That might make investigation into the bug easier.
I'm especially interested in the ping reason, but just the full ping might avoid back-and-forth.

Comment 2

[Jeff Klukas [:klukas] (UTC-4)]

I just sent :janerik a full payload for one of these pings via Slack.

Comment 3

[Jan-Erik Rediger [:janerik]]

Thanks, :klukas.

❯ jq .ping_info.reason focus_40k_events.json
"startup"

As I expected this is a startup ping. That's basically the only way where there's a possibility for the ping to contain more than 500 events.
That still doesn't explain why so many events weren't sent on backgrounding or at max capacity though.

Comment 4

[Jan-Erik Rediger [:janerik]]

The code causing this particular influx of events was changed, so it's not immediately causing problems anymore to my knowledge.
It's still unclear why the Glean SDK is not limiting events correctly. We're going to look into that at some other time.

Comment 5

[Alessio Placitelli [:Dexter]]

(In reply to Jan-Erik Rediger [:janerik] from comment #4)

The code causing this particular influx of events was changed, so it's not immediately causing problems anymore to my knowledge.
It's still unclear why the Glean SDK is not limiting events correctly. We're going to look into that at some other time.

Note that this is relatively important due to the near-future usage of events in custom pings for awesomebar/new tab. Let's revisit this in the next triage cycle.

Comment 6

[Jan-Erik Rediger [:janerik]]

Ack. We should get around to do it, raising priority.

Comment 7

[Jan-Erik Rediger [:janerik]]

While this might still be an existing bug in Glean, it's of less priority now that custom pings don't adhere to that limit anymore (with Glean v52).
Dropping priority.