2.95% raptor-tp6-yahoo-mail-firefox-cold confidence (linux64-shippable-qr) regression on push fcb6e5236bd628dcb6dca78d52eb5df528822a61 (Tue June 30 2020)
Categories
(Core :: Networking: HTTP, defect, P2)
Tracking
()
Tracking | Status | |
---|---|---|
firefox-esr68 | --- | unaffected |
firefox-esr78 | --- | unaffected |
firefox78 | --- | unaffected |
firefox79 | --- | unaffected |
firefox80 | --- | wontfix |
People
(Reporter: Bebe, Assigned: CuveeHsu)
References
(Regression)
Details
(Keywords: perf, perf-alert, regression, Whiteboard: [necko-triaged])
Perfherder has detected a raptor performance regression from push fcb6e5236bd628dcb6dca78d52eb5df528822a61. As author of one of the patches included in that push, we need your help to address this regression.
Regressions:
3% raptor-tp6-yahoo-mail-firefox-cold confidence linux64-shippable-qr opt webrender 74.88 -> 72.67
Improvements:
15% raptor-tp6-slides-firefox-cold loadtime windows10-64-shippable opt 2,289.06 -> 1,935.08
14% raptor-tp6-slides-firefox-cold loadtime windows10-64-shippable-qr opt webrender 2,220.17 -> 1,899.75
13% raptor-tp6-slides-firefox-cold loadtime windows7-32-shippable opt 2,210.12 -> 1,914.17
3% raptor-tp6-slides-firefox-cold windows10-64-shippable-qr opt webrender 785.06 -> 758.44
Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.
For more information on performance sheriffing please see our FAQ.
Reporter | ||
Comment 1•4 years ago
|
||
Even if this can be considered a improvement we have a regression on network replay confidence metric.
Can you take a look over this please
Assignee | ||
Comment 2•4 years ago
|
||
We have bug 1649965 and bug 1646592 for the perf optimization, which should ease the perf regression.
Assignee | ||
Comment 3•4 years ago
|
||
I run ./mach raptor --test yahoo-mail
locally with linux64 debug and I believe bug 1646592+bug 1649965 helps.
subset\revision | central | bug 1646592 | bug 1646592+bug 1649965 |
---|---|---|---|
dcf | 17907.0 | 17682.0 | 11538.0 |
fcp | 3679.5 | 3667.0 | 2664.5 |
fnbpaint | 3593.5 | 3578.5 | 2574.5 |
loadtime | 17952.5 | 17710.0 | 14672.5 |
pageload(ms) | 8074.8 | 8006.79 | 5837.96 |
all of them are lowerIsBetter:true
I'm not sure if it kinda represents raptor-tp6-yahoo-mail-firefox-cold confidence linux64-shippable-qr opt webrender
since I didn't see the number of confidence
in raptor.json. Should we just wait for the optimized bugs given these number, :Bebe?
Comment 4•4 years ago
|
||
Set release status flags based on info from the regressing bug 1633935
Reporter | ||
Comment 5•4 years ago
|
||
confidence is generated in a separate file: perfherder-data-mozproxy.json
Also you run try build with mach try fizzy and select raptor-tp6-yahoo-mail on linux64-shippable-qr
Then we can generate retry and easily see the graphs
Assignee | ||
Comment 6•4 years ago
|
||
Could you provide more information of confidence
.
When I look at the table, fcb6e5236bd doesn't looks like an outstanding regression. It's 73, but the numbers are 73-74 all around the table.
We do have 70 in another commit (6761d9831332f), not sure if it's pretty bad since I need to know more information.
I do mach try fuzzy
in the autoland commit of the bug 1649965, but it's still around 73-74.
It might be an early judgement, but I don't see bug 1633935 is a regressor.
Assignee | ||
Comment 7•4 years ago
|
||
The alert itself is raptor-tp6-yahoo-mail-firefox-cold confidence linux64-shippable-qr opt webrender
and the table I look at in comment 16 is raptor-tp6-yahoo-mail-firefox-cold confidence linux64-shippable-qr opt
(without webrender)
So I need to know more information of confidence
, which is not in the wiki.
Reporter | ||
Comment 8•4 years ago
|
||
confidence
is calculated by the proxy playback tool. we generate from the total requests the browser makes as replayed
and the total not replayed
requests received by the proxy from the browser as:
confidence = float(self._replayed) / float(self._replayed + self._not_replayed)
As we have a historic data on this metric we can detect if the browser generates more or less request for the exact website recording
:tarek please confirm this is correct
Assignee | ||
Comment 9•4 years ago
|
||
How to decide if we want to replayed or not_replayed the response?
I'm not good at the mitmproxy code here
https://searchfox.org/mozilla-central/rev/1b95a0179507a4dc7d4b0c94c2df420dc1a72885/testing/mozbase/mozproxy/mozproxy/backends/mitm/scripts/alternate-server-replay.py#225
Is it about the order of requests in the cold-replay?
The order might be change within spec, but bug 1633935 doesn't change the correctness of response.
Assignee | ||
Comment 10•4 years ago
|
||
close this given I stop the investigation and the confidence value is recovered.
Updated•4 years ago
|
Updated•3 years ago
|
Updated•3 years ago
|
Description
•