1625071 - 64.51% build times (osx-cross) regression on push b82effbcaabc92e7d87f18b00cba192219f479e2 (Tue March 10 2020)

Status: RESOLVED WONTFIX

(Core :: DOM: Security, defect, P1)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=b82effbcaabc92e7d87f18b00cba192219f479e2

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

65% build times osx-cross asan asan-fuzzing taskcluster-c5d.4xlarge 821.46 -> 1,351.40

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=25474

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Comment 1

[Ethan Tseng [:ethan]]

Christoph, this regression looks pretty bad. Could you look into it?

Comment 2

[Christoph Kerschbaumer [:ckerschb]]

(In reply to Ethan Tseng [:ethan] from comment #1)

Christoph, this regression looks pretty bad. Could you look into it?

Basti is on it!

Comment 3

[Sebastian Streich [:sstreich]]

So looking into the patch i sadly could not pinpoint why this would be causing this build regression - as this patch just copies a function from one file into another and moves the calls there.

I could quickly make a patch to remove all calls referencing the introduced function, leaving this a no-op so we can see if the build time improves. Would that be acceptable?

Comment 4

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Yes, please. Let's see what's happening.

Comment 5

[Sebastian Streich [:sstreich]]

Attachment: Bug 1625071 - Revert nsIPrincipal-GetExposableSpec and callers r=ckerschb

Comment 6

[Sebastian Streich [:sstreich]]

Hey!
The patches connected to this were backed out so hopefully we'll see some improvements :)

Bug 1620624
Bug 1620984

--- Backouts
https://hg.mozilla.org/integration/autoland/rev/67b2ab231fe62dd8f29fee54f3f115d722239caa
https://hg.mozilla.org/integration/autoland/rev/f70fcfa75fdd81af32133cf9be5575faee2f7685

Comment 7

[Sebastian Streich [:sstreich]]

Hey, did the build times improve? Just wanted to know if we can close the bug or if there is anything else that's causing problems :)

Comment 8

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Hei. No, it didn't.

Comment 9

[Ethan Tseng [:ethan]]

(In reply to Alexandru Ionescu :alexandrui (needinfo me) from comment #8)

Hei. No, it didn't.

Perhaps the regression wasn't actually caused by bug 1620624?

Comment 10

[Ethan Tseng [:ethan]]

(In reply to Alexandru Ionescu :alexandrui (needinfo me) from comment #8)

Hei. No, it didn't.

Hi Alexandru,
Sebastian backed out his patches of bug 1620624 nine days ago but you said the build time hasn't improved.
We guess bug 1620624 is not the real root cause of this regression.
Do you know how to conduct a further investigation here?

Comment 11

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Hi Ethan,
I'm looking into it. Looks like the only suspect left is Bug 1619641. I'm going to ask there about this. Sorry for the disturbance, the build metrics graph are a special type that can't be re-triggered and they also are unstable, thus they are prone to errors.
Will keep you posted. Thanks!

Comment 12

[Ryan VanderMeulen [:RyanVM]]

I'm looking at that graph and trying to discern how we can see any meaningful regression within all that noise. I'd still take a low-risk patch if there's something to fix, but those odds seem pretty low :\

Comment 13

[Ethan Tseng [:ethan]]

(In reply to Alexandru Ionescu :alexandrui (needinfo me) from comment #11)

Hi Ethan,
I'm looking into it. Looks like the only suspect left is Bug 1619641. I'm going to ask there about this. Sorry for the disturbance, the build metrics graph are a special type that can't be re-triggered and they also are unstable, thus they are prone to errors.
Will keep you posted. Thanks!

Thank you, Alexandru!

Comment 14

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Christoph, you can land back the patch. Thanks.

Comment 15

[Christoph Kerschbaumer [:ckerschb]]

(In reply to Alexandru Ionescu (needinfo me) :alexandrui from comment #14)

Christoph, you can land back the patch. Thanks.

I think all the patches that got backed out did re-land in the meantime. Basti, am I missing something?

Comment 16

[Sebastian Streich [:sstreich]]

Everything we backed out landed again

Comment 17

[Christoph Kerschbaumer [:ckerschb]]

Basti, is there anything else we need to do here or can we close this bug?

Comment 18

[Sebastian Streich [:sstreich]]

Yes i think we can close this, as there is not much we can do right now :/