1179371 - Subsession chain branching must be quantified

Status: RESOLVED INCOMPLETE

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P4)

Description

[Sam Penrose]

(1) Subsessions with more than one descendant subsession must be “rare” (profile branching can happen, but if session chains branch all the time then there is a problem).

- How do we quantify "rare"?

(2) Subsessions must never have more than one ancestor subsession.

Comment 1

[brendan c]

(In reply to Sam Penrose from comment #0)
> (1) Subsessions with more than one descendant subsession must be “rare”
> (profile branching can happen, but if session chains branch all the time
> then there is a problem).
> 
> - How do we quantify "rare"?
> 
> (2) Subsessions must never have more than one ancestor subsession.

Good question. I specifically put the scare quotes on "rare" because this is kind of an open question.

Based on mostly my intuition [1], my opening bid is that we should probably see true branching in less than 1 in 1000 clientIds; if we see evidence of branching more often than this we will want to dig in to a bunch of branching histories to see if they look like plausible histories generated by e.g. testing machines, or if they look like bugs.

One wrinkle is that it's possible we may see more branching on nightly due to our own testing infrastructure. In order to exclude that possibility, I wonder if we should talk to the people that run the test infrastructure. Maybe they can run some kind of script to send us a list of clientIds belonging to FF instances used in testing?

[1] my intuition is somewhat informed by the plots here -- https://metrics.mozilla.com/protected/bcolloran/clientIdsAndLinkedRecords/clientIdsAndLinkedRecords.html -- but I haven't done a proper study of the matter.

Comment 2

[brendan c]

(In reply to Sam Penrose from comment #0)
> (1) Subsessions with more than one descendant subsession must be “rare”
> (profile branching can happen, but if session chains branch all the time
> then there is a problem).
>
> (2) Subsessions must never have more than one ancestor subsession.


Oh, I also note that the terms "ancestor" and "descendant" were not fully explained in my original write-up; in case it wasn't clear:
for subsessions A and B, if A is B's ancestor, then B's previousSubsessionId will point to A's subsessionId


> (2) Subsessions must never have more than one ancestor subsession.

I think this is probably not even possible at all; there would need to be more than one previousSubsessionId entry for in a subsession, which seems unlikely to be possible. I think I must have just put this second criterion in by way of analogy from fhr v2, but I think we can strike it.

Comment 3

[brendan c]

This notebook speaks to the prevalence of subsession chain branching:
http://nbviewer.ipython.org/gist/bcolloran/f2e7ef33989067ecb34d#Percentage-of-clients-with-bad-session-graphs,-and-kinds-of-bad-graphs
(see cell 19)

In the sample used there, we saw about 0.4% of clients having branching.

Benjamin notes here -- https://bugzilla.mozilla.org/show_bug.cgi?id=1171268#c6 -- that it seems reasonable that 0.4% of profiles on Nightly will have branching histories, and I concur with that judgement.

Comment 4

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

Moving per :gfritzsche

Comment 5

[Katie Parlante]

Not prioritized right now. The tools/data are available to do this -- if anyone wants to take this it would be a useful telemetry health metric.

Comment 6

[Firefox Bug Husbandry Bot]

Closing abandoned bugs in this product per https://bugzilla.mozilla.org/show_bug.cgi?id=1337972