1099406 - [e10s][telemetry] regression in SIMPLE_MEASURES_FIRSTLOADURI (aka simpleMeasurements/firstLoadURI)

Status: RESOLVED INVALID

(Core Graveyard :: Tracking, defect, P2)

Description

[Vladan Djeric (:vladan)]

Turning on e10s by default on the 2014-11-07 Nightly caused the time to first load of a URL after browser launch (SIMPLE_MEASURES_FIRSTLOADURI) to regress significantly:

http://alerts.telemetry.mozilla.org/index.html#/detectors/1/metrics/521/alerts/?from=2014-11-09&to=2014-11-09

Comment 1

[Vladan Djeric (:vladan)]

Possible dupe of bug 1096756?

Comment 2

[Jim Mathies [:jimm]]

https://telemetry.mozilla.org/new-pipeline/dist.html#!compare=e10sEnabled&cumulative=0&end_date=2015-11-09&keys=__none__!__none__!__none__&max_channel_version=aurora%252F44&measure=SIMPLE_MEASURES_FIRSTLOADURI&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2015-11-02&table=0&trim=1&use_submission_date=0

Comment 4

[Tracy Walker [:tracy]]

Something landed around Nov. 20/21 that gave a notable improvement for this on Nightly(45) e10s.  But there is still a spike near 7 sec.

Comment 5

[Jim Mathies [:jimm]]

This is still an issue - 

Median difference in startup time is 4562.00, (9173.00, 4611.00).
The probability of this effect being purely by chance is 0.00.

https://github.com/vitillo/e10s_analyses/blob/master/beta46-noapz/e10s_experiment.ipynb

Gabor, would you mind poking at this to see if it's real or maybe just a problem with the probe under e10s? I would expect our startup times to improve here.

Comment 6

[Chris Peterson [:cpeterson]]

The Beta 46 experiment shows a 4.5 second regression in e10s for this simpleMeasurements/firstLoadUri metric. Median difference in startup time is 4562.00, (e10s=9173.00, non-e10s=4611.00).

https://github.com/vitillo/e10s_analyses/blob/master/beta46-noapz/e10s_experiment.ipynb

I have added firstLoadUri to the e10s release criteria wiki's "Startup/Shutdown Time" section:

https://wiki.mozilla.org/Electrolysis/Release_Criteria#Startup.2FShutdown_Time

Comment 7

[Gabor Krizsanits (INACTIVE)]

(In reply to Jim Mathies [:jimm] from comment #5) 
> Gabor, would you mind poking at this to see if it's real or maybe just a
> problem with the probe under e10s? I would expect our startup times to
> improve here.

This probe is quite meaningless for e10s. First, we have this in content process and in the parent process as well and these numbers are just mixed together I think. The one in the content process measures that after launching the content process how much time it takes until we start to load a non about-blank URI. This is _not_ what the non-e10s version measures (there we measure the time from browser startup) and these numbers as expected looks better: https://telemetry.mozilla.org/new-pipeline/dist.html#!compare=e10sEnabled&cumulative=0&e10s=true&end_date=2015-12-14&keys=__none__!__none__!__none__&max_channel_version=aurora%252F44&measure=SIMPLE_MEASURES_FIRSTLOADURI&min_channel_version=null&processType=true&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2015-10-29&table=0&trim=1&use_submission_date=0

In the parent process... It worth nothing that about:blank for example does not count, so people with about:blank as their default homepage will give invalid data. Also we have this trick for new tab that we pre-load it before it was opened, that for some reason does count. But most disturbingly what this probe should measure is when actually load some actual web-content, but in that case the content docshell is in the child process so that case won't work either (so any web content as the startup page).

Oddly we get sometimes a load with empty string, I'm not sure where it comes from but we count that so we have enough data in the parent process as well, and these numbers look worse:

https://telemetry.mozilla.org/new-pipeline/dist.html#!compare=e10sEnabled&cumulative=0&e10s=true&end_date=2015-12-14&keys=__none__!__none__!__none__&max_channel_version=aurora%252F44&measure=SIMPLE_MEASURES_FIRSTLOADURI&min_channel_version=null&processType=false&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2015-10-29&table=0&trim=1&use_submission_date=0

What does that mean? I'd say: not much. Should we block on this? I don't think so, the probe seems to be completely broken. Should we fix the probe and re-do the measurement? Maybe... but I fail to see the usefulness of this probe to be honest. It measures something that has no meaning for a user and work significantly differently in e10s than in non-e10s. The numbers will look differently for better or worse depending on how we define the probe.

Comment 8

[Chris Peterson [:cpeterson]]

(In reply to Gabor Krizsanits [:krizsa :gabor] from comment #7)
> What does that mean? I'd say: not much. Should we block on this? I don't
> think so, the probe seems to be completely broken. Should we fix the probe
> and re-do the measurement? Maybe... but I fail to see the usefulness of this
> probe to be honest. It measures something that has no meaning for a user and
> work significantly differently in e10s than in non-e10s. The numbers will
> look differently for better or worse depending on how we define the probe.

Thanks, Gabor. I'll remove FIRSTLOADURI from the e10s release criteria. It sounds like there are a lot of problems with FIRSTLOADURI that need to be resolved or perhaps the probe should just be removed.

I'll close this bug as RESOLVED INVALID unless you think we should do something else.