Closed Bug 1625071 Opened 5 months ago Closed 3 months ago

64.51% build times (osx-cross) regression on push b82effbcaabc92e7d87f18b00cba192219f479e2 (Tue March 10 2020)

Categories

(Core :: DOM: Security, defect, P1)

defect

Tracking

()

RESOLVED WONTFIX
Tracking Status
firefox-esr68 --- unaffected
firefox75 --- unaffected
firefox76 - wontfix
firefox77 --- wontfix
firefox78 --- wontfix

People

(Reporter: alexandrui, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: perf-alert, regression, Whiteboard: [domsecurity-active])

Attachments

(1 obsolete file)

We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=b82effbcaabc92e7d87f18b00cba192219f479e2

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

65% build times osx-cross asan asan-fuzzing taskcluster-c5d.4xlarge 821.46 -> 1,351.40

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=25474

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Flags: needinfo?(sstreich)
Component: Performance → DOM: Security
Product: Testing → Core
Version: Version 3 → unspecified

Christoph, this regression looks pretty bad. Could you look into it?

Flags: needinfo?(ckerschb)

(In reply to Ethan Tseng [:ethan] from comment #1)

Christoph, this regression looks pretty bad. Could you look into it?

Basti is on it!

Flags: needinfo?(ckerschb)

So looking into the patch i sadly could not pinpoint why this would be causing this build regression - as this patch just copies a function from one file into another and moves the calls there.

I could quickly make a patch to remove all calls referencing the introduced function, leaving this a no-op so we can see if the build time improves. Would that be acceptable?

Flags: needinfo?(sstreich) → needinfo?(aionescu)

Yes, please. Let's see what's happening.

Flags: needinfo?(aionescu)
Assignee: nobody → sstreich
Status: NEW → ASSIGNED
Priority: -- → P1
Whiteboard: [domsecurity-active]
Attachment #9136769 - Attachment is obsolete: true

Hey, did the build times improve? Just wanted to know if we can close the bug or if there is anything else that's causing problems :)

Flags: needinfo?(aionescu)

Hei. No, it didn't.

Flags: needinfo?(aionescu)

(In reply to Alexandru Ionescu :alexandrui (needinfo me) from comment #8)

Hei. No, it didn't.

Perhaps the regression wasn't actually caused by bug 1620624?

(In reply to Alexandru Ionescu :alexandrui (needinfo me) from comment #8)

Hei. No, it didn't.

Hi Alexandru,
Sebastian backed out his patches of bug 1620624 nine days ago but you said the build time hasn't improved.
We guess bug 1620624 is not the real root cause of this regression.
Do you know how to conduct a further investigation here?

Flags: needinfo?(aionescu)

Hi Ethan,
I'm looking into it. Looks like the only suspect left is Bug 1619641. I'm going to ask there about this. Sorry for the disturbance, the build metrics graph are a special type that can't be re-triggered and they also are unstable, thus they are prone to errors.
Will keep you posted. Thanks!

Flags: needinfo?(aionescu)
Assignee: sstreich → nobody
Status: ASSIGNED → NEW
No longer regressed by: 1620624
Target Milestone: mozilla76 → ---

I'm looking at that graph and trying to discern how we can see any meaningful regression within all that noise. I'd still take a low-risk patch if there's something to fix, but those odds seem pretty low :\

(In reply to Alexandru Ionescu :alexandrui (needinfo me) from comment #11)

Hi Ethan,
I'm looking into it. Looks like the only suspect left is Bug 1619641. I'm going to ask there about this. Sorry for the disturbance, the build metrics graph are a special type that can't be re-triggered and they also are unstable, thus they are prone to errors.
Will keep you posted. Thanks!

Thank you, Alexandru!

Christoph, you can land back the patch. Thanks.

Flags: needinfo?(ckerschb)

(In reply to Alexandru Ionescu (needinfo me) :alexandrui from comment #14)

Christoph, you can land back the patch. Thanks.

I think all the patches that got backed out did re-land in the meantime. Basti, am I missing something?

Flags: needinfo?(ckerschb) → needinfo?(sstreich)

Everything we backed out landed again

Flags: needinfo?(sstreich)

Basti, is there anything else we need to do here or can we close this bug?

Flags: needinfo?(sstreich)

Yes i think we can close this, as there is not much we can do right now :/

Status: NEW → RESOLVED
Closed: 3 months ago
Flags: needinfo?(sstreich)
Resolution: --- → FIXED
Resolution: FIXED → WONTFIX
You need to log in before you can comment on or make changes to this bug.