Chromium mac build times out on latest revision

RESOLVED FIXED in Firefox 67

Status

defect
P1
normal
RESOLVED FIXED
5 months ago
2 months ago

People

(Reporter: sparky, Assigned: rwood)

Tracking

Version 3
mozilla68
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox67 fixed, firefox68 fixed)

Details

(Whiteboard: [stockwell disable-recommended])

Attachments

(2 attachments)

The latest chromium mac build on revision "637698" fails on some (not all) benchmark tests with a timeout as can be seen here. The tp6 tests ran with no problem though: https://treeherder.mozilla.org/#/jobs?repo=try&revision=180463d45da964df3f43bbceabab488a5db8d565

Depends on: 1476372

The failing benchmark tests on the latest Chromium are: motionmark-htmlsuite-chrome, stylebench-chrome, speedometer-chrome, sunspider-chrome.

Duplicate of this bug: 1533962
Duplicate of this bug: 1533961

Also webaudio-chrome

Duplicate of this bug: 1534020
Duplicate of this bug: 1533726

Let's let this ride for now and see if a few more automatic updates of Chromium in production will solve this.

Assignee: nobody → rwood
Status: NEW → ASSIGNED
Priority: -- → P1
Whiteboard: [stockwell disable-recommended] → [stockwell needswork]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork]

The failure rate here is still increasing. There are 137 failures in the last 7 days. This is a recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=237301971&repo=mozilla-central&lineNumber=576

Robert, could you please take a look over this? Thank you.

Flags: needinfo?(rwood)
Whiteboard: [stockwell disable-recommended] → [stockwell needswork]
Blocks: 1514477

(In reply to Cosmin Sabou [:CosminS] from comment #20)

The failure rate here is still increasing. There are 137 failures in the last 7 days. This is a recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=237301971&repo=mozilla-central&lineNumber=576

Robert, could you please take a look over this? Thank you.

Yes I will thanks Cosmin, sorry it has taken us a long time to get to this. If I can't get a resolution today then I will submit a patch to temporarily disable these tests (except for on try).

Flags: needinfo?(rwood)
Posted image chromium-failure.png

Ahhhh. Reproduced locally on OSX. It's this keychain error again (see attached screenshot).

I fixed this already - but turns out the fix was only made for page-load tests, not benchmarks:

https://searchfox.org/mozilla-central/rev/44a212460990ffffecf50a8e972d3cbde2e7216b/testing/raptor/raptor/raptor.py#504

I will make a patch that applies that same '--use-mock-keychain' param to benchmarks tests.

looks like try failed.

I also tried to get the patch locally but had issues with hg:

moz-phab patch D25861
Patching revision: D25861
abort: unknown revision 'e8d7542e19fc43a6640cbbb482bdb0e092bd19e3'!

(In reply to Florin Strugariu [:Bebe] from comment #27)

looks like try failed.

Yeah the build on try failed so the raptor jobs didn't even start. I'll rebase and push to try again.

Duplicate of this bug: 1514477

(In reply to Robert Wood [:rwood] from comment #29)

https://treeherder.mozilla.org/#/jobs?repo=try&revision=af998896c8c4191fcdfe5a9387a6a76a83b08697

:jmaher, any idea why this build (OSX cross-compiled shippable opt) would be failing on try but not in production? (Or whom I could ask). I can't seem to run Raptor test jobs on OSX chromium on try. Thanks!

Flags: needinfo?(jmaher)

that looks to be the artifact build broken, possibly we don't have support for shippable build artifacts.

:callek, do you know if the artifact builds were enabled for shippable? If not, we could ask :chmanchester or :mshal to add support.

Flags: needinfo?(jmaher) → needinfo?(bugspam.Callek)

That is https://bugzilla.mozilla.org/show_bug.cgi?id=1540941

I'm looking into that this week

Flags: needinfo?(bugspam.Callek)
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

:davehunt, can you look into this- we will be disabling this job in the short term if there isn't a fix soon.

Flags: needinfo?(dave.hunt)
Pushed by dhunt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/09e7dea38c39
Fix Raptor Chromium OSX jobs by using mock-keychain; r=davehunt

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #40)

:davehunt, can you look into this- we will be disabling this job in the short term if there isn't a fix soon.

The fix hasn't changed in the last week, it was simply blocked by bug 1540941. My latest try run was successful so I've queued the patch for landing.

Flags: needinfo?(dave.hunt)
Whiteboard: [stockwell disable-recommended] → [stockwell needswork]
Status: ASSIGNED → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla68

Yes, this should be uplifted to mozilla-beta to fix the perma-failure.

Flags: needinfo?(dave.hunt)
Whiteboard: [stockwell disable-recommended] → [stockwell disable-recommended][checkin-needed-beta]
Whiteboard: [stockwell disable-recommended][checkin-needed-beta]
You need to log in before you can comment on or make changes to this bug.