Closed Bug 1651072 Opened 4 years ago Closed 4 years ago

2.95% raptor-tp6-yahoo-mail-firefox-cold confidence (linux64-shippable-qr) regression on push fcb6e5236bd628dcb6dca78d52eb5df528822a61 (Tue June 30 2020)

Categories

(Core :: Networking: HTTP, defect, P2)

80 Branch
defect

Tracking

()

RESOLVED WONTFIX
Tracking Status
firefox-esr68 --- unaffected
firefox-esr78 --- unaffected
firefox78 --- unaffected
firefox79 --- unaffected
firefox80 --- wontfix

People

(Reporter: Bebe, Assigned: CuveeHsu)

References

(Regression)

Details

(Keywords: perf, perf-alert, regression, Whiteboard: [necko-triaged])

Perfherder has detected a raptor performance regression from push fcb6e5236bd628dcb6dca78d52eb5df528822a61. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

3% raptor-tp6-yahoo-mail-firefox-cold confidence linux64-shippable-qr opt webrender 74.88 -> 72.67

Improvements:

15% raptor-tp6-slides-firefox-cold loadtime windows10-64-shippable opt 2,289.06 -> 1,935.08
14% raptor-tp6-slides-firefox-cold loadtime windows10-64-shippable-qr opt webrender 2,220.17 -> 1,899.75
13% raptor-tp6-slides-firefox-cold loadtime windows7-32-shippable opt 2,210.12 -> 1,914.17
3% raptor-tp6-slides-firefox-cold windows10-64-shippable-qr opt webrender 785.06 -> 758.44

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Even if this can be considered a improvement we have a regression on network replay confidence metric.
Can you take a look over this please

Component: Performance → Networking: HTTP
Flags: needinfo?(juhsu)
Product: Testing → Core

We have bug 1649965 and bug 1646592 for the perf optimization, which should ease the perf regression.

Assignee: nobody → juhsu
Depends on: 1649965, 1646592
Flags: needinfo?(juhsu)
Priority: -- → P2
Whiteboard: [necko-triaged]

I run ./mach raptor --test yahoo-mail locally with linux64 debug and I believe bug 1646592+bug 1649965 helps.

subset\revision central bug 1646592 bug 1646592+bug 1649965
dcf 17907.0 17682.0 11538.0
fcp 3679.5 3667.0 2664.5
fnbpaint 3593.5 3578.5 2574.5
loadtime 17952.5 17710.0 14672.5
pageload(ms) 8074.8 8006.79 5837.96

all of them are lowerIsBetter:true

I'm not sure if it kinda represents raptor-tp6-yahoo-mail-firefox-cold confidence linux64-shippable-qr opt webrender since I didn't see the number of confidence in raptor.json. Should we just wait for the optimized bugs given these number, :Bebe?

Flags: needinfo?(fstrugariu)

Set release status flags based on info from the regressing bug 1633935

confidence is generated in a separate file: perfherder-data-mozproxy.json
Also you run try build with mach try fizzy and select raptor-tp6-yahoo-mail on linux64-shippable-qr
Then we can generate retry and easily see the graphs

Flags: needinfo?(fstrugariu)

Could you provide more information of confidence.
When I look at the table, fcb6e5236bd doesn't looks like an outstanding regression. It's 73, but the numbers are 73-74 all around the table.

We do have 70 in another commit (6761d9831332f), not sure if it's pretty bad since I need to know more information.

I do mach try fuzzy in the autoland commit of the bug 1649965, but it's still around 73-74.
It might be an early judgement, but I don't see bug 1633935 is a regressor.

Flags: needinfo?(fstrugariu)

The alert itself is raptor-tp6-yahoo-mail-firefox-cold confidence linux64-shippable-qr opt webrender and the table I look at in comment 16 is raptor-tp6-yahoo-mail-firefox-cold confidence linux64-shippable-qr opt (without webrender)

So I need to know more information of confidence, which is not in the wiki.

confidence is calculated by the proxy playback tool. we generate from the total requests the browser makes as replayed and the total not replayed requests received by the proxy from the browser as:
confidence = float(self._replayed) / float(self._replayed + self._not_replayed)

As we have a historic data on this metric we can detect if the browser generates more or less request for the exact website recording
:tarek please confirm this is correct

Flags: needinfo?(fstrugariu) → needinfo?(tarek)
See Also: → 1650003

How to decide if we want to replayed or not_replayed the response?
I'm not good at the mitmproxy code here
https://searchfox.org/mozilla-central/rev/1b95a0179507a4dc7d4b0c94c2df420dc1a72885/testing/mozbase/mozproxy/mozproxy/backends/mitm/scripts/alternate-server-replay.py#225
Is it about the order of requests in the cold-replay?
The order might be change within spec, but bug 1633935 doesn't change the correctness of response.

close this given I stop the investigation and the confidence value is recovered.

Status: NEW → RESOLVED
Closed: 4 years ago
Flags: needinfo?(tarek)
Resolution: --- → WORKSFORME
Resolution: WORKSFORME → WONTFIX
Has Regression Range: --- → yes
You need to log in before you can comment on or make changes to this bug.