1859614 - Suggestions for metric incident investigation playbook

Status: RESOLVED FIXED

(Data Platform and Tools :: Glean: SDK, enhancement)

Description

[Travis Long [:travis_]]

This bug is aimed mainly at Frank and the work on this doc: https://docs.google.com/document/d/1tmI7PyHR1DGjbyTU8J1hAMq0lwngm6cV9c6E8SppCFw/edit

I had some additional (and some overlapping) suggestions that I have found useful, along with some explanations of what each split might tell us if we find anything:

Countries (e.g. China, Iran..?)

• If so, is there a national holiday or something similar going on?
• Is this an area known for bots or unusual activity (malaysia, china, Ireland, etc.)

ISP (e.g. BrowserStack?)

• Typically this is more fine grained than country and can be more proof of potential bots or automation if the anomaly is coming from a single ISP.
• There’s a lot of ISPs, might need a HAVING clause to filter out smaller ISPs

Product Version/build-id

• Did this start in a specific product version, if so, what changed in that version (work with the product team to answer this question)?
• Is this build-id a known Mozilla build-id, if not it could be a clone/fork or sideload build.

Glean SDK version

• Did this start in a new Glean version, if so, what changed in that version (work with the Glean team to answer this question)?

Other library version changes?

• Check Application Services updates, gecko updates, etc. to see if this can be tied to a specific version change there. Glean relies on things like viaduct and rkv which can affect data collection if there is a regression.

OS-SDK version (android SDK, iOS targets, etc)

• Something may have changed in the platform SDK that is affecting data collection.
• Typically this behavior shows up in either platform lifecycle event behavior changing (see 0 duration pings, etc), or in background task work (uploading pings)

Time difference between start/end_time and submission_timestamp.

• Do the timestamps we record appear reasonable for both the ping time window and the delay from collection/submission to receiving the ping in ingestion.

What Glean errors are there?

• Any networking or other telemetry errors that might be indicative of the issue? This could be an ingestion issue, etc.

Hardware mfg./version/etc.

• Does this only happen on older/newer hardware?

Comment 1

[Travis Long [:travis_]]

https://mozilla.github.io/glean/book/user/howto/investigating-data-issues/investigating-data-issues.html

This was the result of this work seen in the link above.