1533008 - Chromium mac build times out on latest revision

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P1)

Description

[Greg Mierzwinski [:sparky]]

The latest chromium mac build on revision "637698" fails on some (not all) benchmark tests with a timeout as can be seen here. The tp6 tests ran with no problem though: https://treeherder.mozilla.org/#/jobs?repo=try&revision=180463d45da964df3f43bbceabab488a5db8d565

Comment 1

[Robert Wood [:rwood]]

The failing benchmark tests on the latest Chromium are: motionmark-htmlsuite-chrome, stylebench-chrome, speedometer-chrome, sunspider-chrome.

Comment 4

[Robert Wood [:rwood]]

Also webaudio-chrome

Comment 7

[Intermittent Failures Robot]

15 failures in 513 pushes (0.029 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 9
• try: 2
• mozilla-beta: 4

Platform breakdown:

• macosx64-nightly: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-03-12&endday=2019-03-12&tree=all

Comment 8

[Intermittent Failures Robot]

15 failures in 524 pushes (0.029 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 7
• try: 6
• mozilla-beta: 2

Platform breakdown:

• macosx64-nightly: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-03-14&endday=2019-03-14&tree=all

Comment 9

[Intermittent Failures Robot]

20 failures in 436 pushes (0.046 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 16
• try: 2
• mozilla-beta: 2

Platform breakdown:

• macosx64-nightly: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-03-15&endday=2019-03-15&tree=all

Comment 10

[Intermittent Failures Robot]

17 failures in 167 pushes (0.102 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 15
• try: 2

Platform breakdown:

• macosx64-nightly: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-03-16&endday=2019-03-16&tree=all

Comment 11

[Intermittent Failures Robot]

97 failures in 2771 pushes (0.035 failures/push) were associated with this bug in the last 7 days.

This is the #18 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• try: 17
• mozilla-beta: 10
• mozilla-central: 70

Platform breakdown:

• macosx64-nightly: 97

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-03-11&endday=2019-03-17&tree=all

Comment 12

[Robert Wood [:rwood]]

Let's let this ride for now and see if a few more automatic updates of Chromium in production will solve this.

Comment 13

[Intermittent Failures Robot]

30 failures in 413 pushes (0.073 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 26
• try: 2
• mozilla-beta: 2

Platform breakdown:

• macosx64-nightly: 30

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-03-18&endday=2019-03-18&tree=all

Comment 14

[Intermittent Failures Robot]

15 failures in 512 pushes (0.029 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 12
• mozilla-beta: 2
• try: 1

Platform breakdown:

• macosx64-nightly: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-03-19&endday=2019-03-19&tree=all

Comment 15

[Intermittent Failures Robot]

89 failures in 2561 pushes (0.035 failures/push) were associated with this bug in the last 7 days.

This is the #27 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• try: 8
• mozilla-central: 69
• mozilla-beta: 12

Platform breakdown:

• macosx64-nightly: 89

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-03-18&endday=2019-03-24&tree=all

Comment 16

[Intermittent Failures Robot]

25 failures in 960 pushes (0.026 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-beta: 5
• mozilla-central: 20

Platform breakdown:

• macosx64-nightly: 5
• osx-10-10-shippable: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-03-28&endday=2019-03-28&tree=all

Comment 17

[Intermittent Failures Robot]

40 failures in 660 pushes (0.061 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 40

Platform breakdown:

• osx-10-10-shippable: 40

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-03-29&endday=2019-03-29&tree=all

Comment 18

[Intermittent Failures Robot]

20 failures in 184 pushes (0.109 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 16
• mozilla-beta: 4

Platform breakdown:

• osx-10-10-shippable: 16
• macosx64-nightly: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-03-30&endday=2019-03-30&tree=all

Comment 19

[Intermittent Failures Robot]

125 failures in 3910 pushes (0.032 failures/push) were associated with this bug in the last 7 days.

This is the #16 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-beta: 20
• mozilla-central: 105

Platform breakdown:

• macosx64-nightly: 24
• osx-10-10-shippable: 101

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-03-25&endday=2019-03-31&tree=all

Comment 20

[Cosmin Sabou [:CosminS]]

The failure rate here is still increasing. There are 137 failures in the last 7 days. This is a recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=237301971&repo=mozilla-central&lineNumber=576

Robert, could you please take a look over this? Thank you.

Comment 21

[Intermittent Failures Robot]

21 failures in 398 pushes (0.053 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 21

Platform breakdown:

• osx-10-10-shippable: 21

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-04-01&endday=2019-04-01&tree=all

Comment 22

[Robert Wood [:rwood]]

(In reply to Cosmin Sabou [:CosminS] from comment #20)

The failure rate here is still increasing. There are 137 failures in the last 7 days. This is a recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=237301971&repo=mozilla-central&lineNumber=576

Robert, could you please take a look over this? Thank you.

Yes I will thanks Cosmin, sorry it has taken us a long time to get to this. If I can't get a resolution today then I will submit a patch to temporarily disable these tests (except for on try).

Comment 23

[Robert Wood [:rwood]]

Attachment: chromium-failure.png

Ahhhh. Reproduced locally on OSX. It's this keychain error again (see attached screenshot).

I fixed this already - but turns out the fix was only made for page-load tests, not benchmarks:

https://searchfox.org/mozilla-central/rev/44a212460990ffffecf50a8e972d3cbde2e7216b/testing/raptor/raptor/raptor.py#504

I will make a patch that applies that same '--use-mock-keychain' param to benchmarks tests.

Comment 24

[Robert Wood [:rwood]]

Attachment: Bug 1533008 - Fix Raptor Chromium OSX jobs by using mock-keychain; r?bebe,davehunt

Comment 25

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d4c9b35b8ba33edb53a22629f1ec77e88d813a89

Comment 26

[Intermittent Failures Robot]

30 failures in 461 pushes (0.065 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 28
• mozilla-beta: 2

Platform breakdown:

• osx-10-10-shippable: 28
• macosx64-nightly: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-04-02&endday=2019-04-02&tree=all

Comment 27

[Florin Strugariu [:Bebe]]

looks like try failed.

I also tried to get the patch locally but had issues with hg:

moz-phab patch D25861
Patching revision: D25861
abort: unknown revision 'e8d7542e19fc43a6640cbbb482bdb0e092bd19e3'!

Comment 28

[Robert Wood [:rwood]]

(In reply to Florin Strugariu [:Bebe] from comment #27)

looks like try failed.

Yeah the build on try failed so the raptor jobs didn't even start. I'll rebase and push to try again.

Comment 29

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=af998896c8c4191fcdfe5a9387a6a76a83b08697

Comment 31

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #29)

https://treeherder.mozilla.org/#/jobs?repo=try&revision=af998896c8c4191fcdfe5a9387a6a76a83b08697

:jmaher, any idea why this build (OSX cross-compiled shippable opt) would be failing on try but not in production? (Or whom I could ask). I can't seem to run Raptor test jobs on OSX chromium on try. Thanks!

Comment 32

[Joel Maher ( :jmaher ) (UTC -8)]

that looks to be the artifact build broken, possibly we don't have support for shippable build artifacts.

:callek, do you know if the artifact builds were enabled for shippable? If not, we could ask :chmanchester or :mshal to add support.

Comment 33

[Justin Wood (:Callek)]

That is https://bugzilla.mozilla.org/show_bug.cgi?id=1540941

I'm looking into that this week

Comment 34

[Intermittent Failures Robot]

20 failures in 638 pushes (0.031 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 15
• mozilla-beta: 5

Platform breakdown:

• osx-10-10-shippable: 15
• macosx64-nightly: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-04-03&endday=2019-04-03&tree=all

Comment 35

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Bug 1540941 has been resolved, so I rebased and pushed to try:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=dd66b7f3a166550749a371166762963a7de167ce

Comment 36

[Intermittent Failures Robot]

34 failures in 574 pushes (0.059 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 29
• mozilla-beta: 5

Platform breakdown:

• osx-10-10-shippable: 27
• macosx64-nightly: 5
• windows7-32-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-04-04&endday=2019-04-04&tree=all

Comment 37

[Intermittent Failures Robot]

40 failures in 561 pushes (0.071 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 35
• mozilla-beta: 5

Platform breakdown:

• osx-10-10-shippable: 31
• windows10-64-shippable: 3
• macosx64-nightly: 5
• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-04-05&endday=2019-04-05&tree=all

Comment 38

[Intermittent Failures Robot]

18 failures in 114 pushes (0.158 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 18

Platform breakdown:

• osx-10-10-shippable: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-04-06&endday=2019-04-06&tree=all

Comment 39

[Intermittent Failures Robot]

183 failures in 2843 pushes (0.064 failures/push) were associated with this bug in the last 7 days.

This is the #10 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 163
• mozilla-beta: 20

Platform breakdown:

• osx-10-10-shippable: 154
• macosx64-nightly: 20
• windows7-32-shippable: 4
• windows10-64-shippable: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-04-01&endday=2019-04-07&tree=all

Comment 40

[Joel Maher ( :jmaher ) (UTC -8)]

:davehunt, can you look into this- we will be disabling this job in the short term if there isn't a fix soon.

Comment 41

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/09e7dea38c39
Fix Raptor Chromium OSX jobs by using mock-keychain; r=davehunt

Comment 42

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #40)

:davehunt, can you look into this- we will be disabling this job in the short term if there isn't a fix soon.

The fix hasn't changed in the last week, it was simply blocked by bug 1540941. My latest try run was successful so I've queued the patch for landing.

Comment 43

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/09e7dea38c39

Comment 44

[Intermittent Failures Robot]

16 failures in 560 pushes (0.029 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-beta: 5
• mozilla-central: 11

Platform breakdown:

• macosx64-nightly: 5
• osx-10-10-shippable: 10
• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-04-08&endday=2019-04-08&tree=all

Comment 45

[Andreea Pavel [:apavel]]

This is perma-failing on mozilla-beta, however I'm not sure why orange factor does not show the classifications made on mozilla-beta

Treeherder link: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception&fromchange=6fb04d1496a957a936558913a2a6dcca38e28537&selectedJob=239370482&searchStr=os%2Cx%2C10.10%2Cnightly%2Copt%2Craptor%2Cperformance%2Ctests%2Con%2Cgoogle%2Cchrome%2Cwith%2Ce10s%2Ctest-macosx64-nightly%2Fopt-raptor-webaudio-chrome-e10s%2Crap-c-e10s%28wa%29

Should the fix here be uplifted?

Comment 46

[Intermittent Failures Robot]

31 failures in 3025 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 19
• mozilla-central: 12

Platform breakdown:

• macosx64-nightly: 19
• osx-10-10-shippable: 10
• windows7-32-shippable: 1
• windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-04-08&endday=2019-04-14&tree=all

Comment 47

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Yes, this should be uplifted to mozilla-beta to fix the perma-failure.

Comment 48

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/releases/mozilla-beta/rev/75a92a484e17

Comment 49

[Intermittent Failures Robot]

2 failures in 2673 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-beta: 2

Platform breakdown:

• macosx64-nightly: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1533008&startday=2019-04-15&endday=2019-04-21&tree=all