1646689 - 2.01 - 4.86% raptor-tp6m-allrecipes-geckoview-cold loadtime / raptor-tp6m-cnn-geckoview-cold confidence (android-hw-g5-7-0-arm7-api-16-shippable) regression on push 567513cee4895872b1e7fb53fbd2793593d11f97 (Fri June 5 2020)

Status: RESOLVED WONTFIX

(Core :: Networking, defect, P2)

Description

[Florin Strugariu [:Bebe]]

Perfherder has detected a raptor performance regression from push cc8881b3128c07537ad0c9c0a0dd35b0d9ce8a99. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

5% raptor-tp6m-allrecipes-geckoview-cold loadtime android-hw-g5-7-0-arm7-api-16-shippable opt 6,571.96 -> 6,891.42
2% raptor-tp6m-cnn-geckoview-cold confidence android-hw-g5-7-0-arm7-api-16-shippable opt 85.04 -> 83.33

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1643042

Comment 2

[Kartikaya Gupta (email:[e-mail])]

If I click on the graph links on the alert page, I see the two regressions, but they are from different things. The first one here shows a regression on June 8 which is three days after bug 1643042 landed. The other one here seems like a more reasonable time period but I will look closer.

Comment 3

[Kartikaya Gupta (email:[e-mail])]

So it's not clear to me from the perfherder data that my patch actually caused the regression. It's certainly noisy but perfherder itself is blaming a different changeset, if you look here you can see it's highlighted rev 567513cee4895872b1e7fb53fbd2793593d11f97 which is bug 1641496.

The metric that's regressing is a loadtime metric which I think is pretty unlikely to be regressed by my patch. That should really only affect some kinds of scrolls. Bug 1641496 seems to do with network loading code and seems like a more plausible regressor.

Comment 4

[Florin Strugariu [:Bebe]]

My mistake I reassigned the alert to the right commit
See alert: https://treeherder.mozilla.org/perf.html?#/alerts?id=26303

Comment 5

[Kartikaya Gupta (email:[e-mail])]

Reassigning bug based on new regressor best-guess.

Comment 6

[Kershaw Chang [:kershaw]]

I'll take a look.

Comment 7

[Kershaw Chang [:kershaw]]

It's still unclear to me why bug 1641496 is the culprit of this.
Actually, socket process is disabled on android, so I think bug 1641496 should not affect the performance on android at all.

I also have trouble on reproducing this on try. I can't find raptor-tp6m jobs with ./mach try fuzzy --full locally.
Florin, do you probably know how to trigger this job on try?

Comment 8

[Kartikaya Gupta (email:[e-mail])]

(In reply to Kershaw Chang [:kershaw] from comment #7)

It's still unclear to me why bug 1641496 is the culprit of this.

It's possible that it's not. I was looking at the graphs which are pretty noisy and it's not clear what the regressing changeset is. However to me yours seemed more likely than my patch since the test seemed to do with loadtime and your patches were affecting network stuff. But maybe we should go back to the graph and try to find more likely culprits.

Comment 9

[Florin Strugariu [:Bebe]]

(In reply to Kershaw Chang [:kershaw] from comment #7)

I also have trouble on reproducing this on try. I can't find raptor-tp6m jobs with ./mach try fuzzy --full locally.
Florin, do you probably know how to trigger this job on try?

it works but you will need to checkout a patch before Jun 10. We migrated all the mobile tests to browsertime and removed the raptor tests.

Comment 10

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #9)

(In reply to Kershaw Chang [:kershaw] from comment #7)

I also have trouble on reproducing this on try. I can't find raptor-tp6m jobs with ./mach try fuzzy --full locally.
Florin, do you probably know how to trigger this job on try?

it works but you will need to checkout a patch before Jun 10. We migrated all the mobile tests to browsertime and removed the raptor tests.

:bebe is it still possible to trigger these (raptor) tests via try? alternatively, can we see if the regression was also noticed in browsertime?

Comment 11

[Florin Strugariu [:Bebe]]

yes you can trigger this via try build if you are on the any commit before:
ab74bac09a68daab7a7e94ad5ffd6b476a704777 (Jun 10 03:42:54 UTC)

Comment 12

[Kershaw Chang [:kershaw]]

(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #11)

yes you can trigger this via try build if you are on the any commit before:
ab74bac09a68daab7a7e94ad5ffd6b476a704777 (Jun 10 03:42:54 UTC)

I think maybe it's not worth to fix this, since we need to use the build before June 10 to reproduce this.
:Bebe, what do you think?

Comment 13

[Florin Strugariu [:Bebe]]

:kershaw this is not my decision maybe someone from releng can approve this

Comment 14

[Kershaw Chang [:kershaw]]

Hi Ryan,

I am not sure if you are the right one to ask, if not, could you forward this to someone? Thanks.
My reasons for not fixing this bug are:

1. It's still unclear that bug 1641496 is the root cause of this regression. Note that socket process is disabled on android, so bug 1641496 should not affect the performance.
2. It's difficult to reproduce this, since raptor-tp6m jobs can only be triggered with the commit before June 10.

What do you think?

Comment 15

[Ryan VanderMeulen [:RyanVM]]

Not sure we have much choice but to accept the regression if we can't easily reproduce the test environment it was reported from anymore.

Comment 16

[Kershaw Chang [:kershaw]]

Close this bug since it's not able to reproduce on the latest nightly.