1834620 - AboutWelcome and Spotlight browser-chrome tests have unstable ordering

Status: NEW

(Firefox :: Messaging System, task, P3)

Description

[Chris H-C :chutten]

We had some difficulty writing tests for bug 1825863 as we expected the ordering of events (IMPRESSION, CLICK, DISMISS, etc) to be consistent. The order that AWPage:TELEMETRY_EVENT events comes in isn't stable from local development to CI or from normal test runs to --verify or sometimes even between runs within the same environment.

This bug is about diagnosing if the event order is unstable in the real world as well as in tests (which should be discernable from the instrumentation added in bug 1825863 (hint: use ping_info.seq to uncover the order)), determining if it can or should be made stable within tests, and adjusting the tests should the order be found to be able to be made stable.

(( Please coordinate with the PingCentre Replacement Project (find :chutten, he's normally around) before landing changes to instrumentation. ))

Comment 1

[Chris H-C :chutten]

Also: don't forget to un-skip-if browser_multistage_spotlight_telemetry.js for verify runs should we find ourselves able to stabilize the order.

Comment 2

[Dan Mosedale (:dmosedale, :dmose)]

The first step, I think, will be to look at the collected telemetry from that bug once it lands...