Closed
Bug 1316835
Opened 8 years ago
Closed 8 years ago
Validate the URI loads triggered by search (Beta Channel)
Categories
(Toolkit :: Telemetry, defect, P1)
Toolkit
Telemetry
Tracking
()
RESOLVED
FIXED
Tracking | Status | |
---|---|---|
firefox52 | --- | affected |
People
(Reporter: Dexter, Assigned: Dexter)
References
Details
(Whiteboard: [measurement:client])
In bug 1308705 we validated the search probes landed with bug 1303333. Since we uplifted the probes, we should cross check that they behave correctly on Beta.
Here's the notebook and SQL query used on Nightly:
[1] - https://gist.github.com/Dexterp37/078e254d3e12d57bf45d5417504d9b71
[2] - https://sql.telemetry.mozilla.org/queries/1612
We also want to cross check that the SEARCH_COUNTS and search scalar probes are behaving similarly.
Assignee | ||
Updated•8 years ago
|
Assignee | ||
Updated•8 years ago
|
Priority: P2 → P1
Assignee | ||
Updated•8 years ago
|
Priority: P1 → P2
Assignee | ||
Comment 1•8 years ago
|
||
Now that we have Beta data available, I took some time to run the validation notebook that we used for Nightly on Beta [1].
Key highlights:
- Only 23% of the pings report search engagement data on Beta (see cell 7).
- SEARCH_COUNTS follow the same trend (see cell 8).
- This is somehow consistent with the data from Beta before the engagement patches landed (see cell 10)
- We've got matching SEARCH_COUNTS and search engagement probes for over 99% of the pings (see cell 14)
The mismatching values are due to the one-off search regression that was introduced by the search engagement patches and fixed in Beta 3.
@Brendan is there anything else to do here?
[1] - https://gist.github.com/Dexterp37/346226b6b4b148cc5d24e732a7ceaafe
Assignee: nobody → alessio.placitelli
Status: NEW → ASSIGNED
Flags: needinfo?(bcolloran)
Priority: P2 → P1
Looks good, thanks Alessio. My only question is, beneath cell [17] you said "It looks that almost 76% of the pings don't have the SEARCH_COUNTS histogram nor the search engagement measurements. That's odd but... seems expected or, at least, not a problem in the counts." -- did you just mean it seems odd because it seems like more pings should have associated searches? Or did it seem like incorrect behavior (e.g., expecting to see the histogram even in the case of zero searches)?
[[[
I don't think that this would make a big difference to the conclusion, just a note on a typo in cell [11] -- i think you meant "if len(search_counts.keys()) < 1:". As entered, "if search_counts.keys() < 1:" will always evaluate to false b/c search_counts.keys() returns a list.
"Note that comparing objects of different types is legal. The outcome is deterministic but arbitrary: the types are ordered by their name. Thus, a list is always smaller than a string, a string is always smaller than a tuple, etc." (see: https://docs.python.org/2/tutorial/datastructures.html#comparing-sequences-and-other-types)
]]]
Flags: needinfo?(bcolloran)
Assignee | ||
Comment 3•8 years ago
|
||
(In reply to brendan c from comment #2)
> Looks good, thanks Alessio. My only question is, beneath cell [17] you said
> "It looks that almost 76% of the pings don't have the SEARCH_COUNTS
> histogram nor the search engagement measurements. That's odd but... seems
> expected or, at least, not a problem in the counts." -- did you just mean it
> seems odd because it seems like more pings should have associated searches?
> Or did it seem like incorrect behavior (e.g., expecting to see the histogram
> even in the case of zero searches)?
Yes, sorry for being cryptic there! I meant the former, I would have expected more pings to have associated search data. But it's a personal belief disproved by the data (and that roughly matches what we saw in Nightly and in previous builds). So no, I don't think there's an incorrect behaviour lurking in the shadows!
> [[[
> I don't think that this would make a big difference to the conclusion, just
> a note on a typo in cell [11] -- i think you meant "if
> len(search_counts.keys()) < 1:". As entered, "if search_counts.keys() < 1:"
> will always evaluate to false b/c search_counts.keys() returns a list.
> "Note that comparing objects of different types is legal. The outcome is
> deterministic but arbitrary: the types are ordered by their name. Thus, a
> list is always smaller than a string, a string is always smaller than a
> tuple, etc." (see:
> https://docs.python.org/2/tutorial/datastructures.html#comparing-sequences-
> and-other-types)
> ]]]
Good catch! Yes, that doesn't change the results much. I ran the notebook again, the results still stand.
Brendan, can we go on and close this bug?
Flags: needinfo?(bcolloran)
Thanks for the clarification Alessio. Yes, let's go ahead and close it out.
Flags: needinfo?(bcolloran)
Assignee | ||
Updated•8 years ago
|
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•