Bugzilla

I believe we want to go with 3.  My rationale is that the probe is intended to measure the time it takes from a user to enter character to getting response from server. 

* Assumption: The time it takes for this process to complete is independent as to whether a suggestion is found. i.e., the latency distribution of responses containing suggestions, is similar (enough) to those responses empty of suggestions. Is this true, or is my assessment naive? 

The comparison is how much latency using Merino adds compared to RemoteSettings, since Merino is being used to replace RemoteSettings. If they are both processing the same search queries, and the assumption above holds, then differences in the distributions informs how Merino adds latency to suggestions being served relative to RemoteSettings. This difference can then be compared to user studies to infer its impact on user perceived performance.

I believe we want to go with 3.  My rationale is that the probe is intended to measure the time it takes from a user to enter character to getting response from server. 

* Assumption: The time it takes for this process to complete is independent as to whether a suggestion is found. i.e., the latency distribution of responses containing suggestions, is similar (enough) to those responses empty of suggestions. Is this true, or is my assessment naive? 

The comparison is how much latency using Merino adds compared to RemoteSettings, since Merino is being used to replace RemoteSettings. If they are both processing the same search queries, and the assumption above holds, then differences in the distributions informs how Merino adds latency to suggestions being served relative to RemoteSettings. This difference can then be compared to user studies to infer its impact on user perceived performance.

I agree in that it gets complicated making that inference as to user perceived performance, given how the results are shown, etc...