Network Predictor Performance Regressions in Nightly 2016-11-19

RESOLVED FIXED in Firefox 53



2 years ago
2 years ago


(Reporter: chutten, Assigned: nwgh)



Firefox Tracking Flags

(firefox53 fixed)


(Whiteboard: [necko-active])

MozReview Requests


Submitter Diff Changes Open Issues Last Updated
Error loading review requests:


(1 attachment)



2 years ago
The Telemetry Alerts system detected two changes in network predictor probes. One for PREDICTOR_PREDICT_TIME_TO_ACTION[1] and another for PREDICTOR_PREDICT_WORK_TIME[2].

The changes appear to be slight performance regressions where the 75th percentile has risen from 0ms to 2ms, a level it hasn't been since June 16th:

The changeset for that build[3] doesn't contain the words "predict" or "network" so I'm at a bit of a loss to narrow down what might have caused it.

Action Items:
1) Determine what caused the regression
2) Figure out if that's expected, or if it needs to be remedied
3) Add "alert_emails" fields to the two probes.

(( The alert_emails field[4] should contain an email address (usually for a mailing list of persons who would like to be notified when the probe changes). ))

Flags: needinfo?(hurley)

Comment 1

2 years ago
So, taking a look at the pushlog, the only thing in there that looks like a plausible culprit to me is the rusturl stuff (the predictor does do a fair amount of url creating). Valentin, do you think the rusturl stuff could have had this impact? Otherwise, I'm at a loss.
Flags: needinfo?(hurley) → needinfo?(valentin.gosu)
Yes, the rusturl did introduce a significant performance issue, but that should be fixed following bug 1318432 (we turned it off via a pref).
Chris, is the regression still present in the telemetry?
Flags: needinfo?(valentin.gosu) → needinfo?(chutten)

Comment 3

2 years ago
The patch in that bug should show up in 20161123

Unfortunately we tried switching something on the backend around then which has since been backed out (bug 1286868). We're backfilling, but for now the aggregate data behind isn't there for the period in question.

I'll leave my ni? up so I remember to look at this tomorrow (~20 hours left in the backfill job, I'm told) so I can give you a better answer then.
Whiteboard: [necko-active]

Comment 4

2 years ago
Yup! This went away ( ).

All that's left is to put appropriate email information in the alert_emails fields of the two probes. There are lots of examples in Histograms.json you can follow. Do you have a mailing list address that would be appropriate?
Flags: needinfo?(chutten) → needinfo?(valentin.gosu)
We have been using
Flags: needinfo?(valentin.gosu)
Assignee: nobody → hurley
Comment hidden (mozreview-request)

Comment 7

2 years ago
Comment on attachment 8823346 [details]
Bug 1320140 - add reporting emails for some predictor telemetry.
Attachment #8823346 - Flags: review?(valentin.gosu) → review+
Comment hidden (mozreview-request)

Comment 9

2 years ago
Pushed by
add reporting emails for some predictor telemetry. r=valentin

Comment 10

2 years ago
Last Resolved: 2 years ago
status-firefox53: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla53
You need to log in before you can comment on or make changes to this bug.