Frequent b2g hazard and device build "timed out after 600 seconds of no output" running and rerunning config.sh

RESOLVED WONTFIX

Status

Release Engineering
General Automation
RESOLVED WONTFIX
2 years ago
9 months ago

People

(Reporter: philor, Unassigned)

Tracking

unspecified

Firefox Tracking Flags

(firefox46 fixed)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

(Reporter)

Description

2 years ago
https://treeherder.mozilla.org/logviewer.html#?job_id=18860428&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=18854765&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=18857974&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=18859002&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=18859463&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=18863098&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=18863220&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=3599478&repo=b2g-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=3599477&repo=b2g-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=3599244&repo=b2g-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=3599468&repo=b2g-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=3598873&repo=b2g-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=3598130&repo=b2g-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=3592844&repo=b2g-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=2937868&repo=mozilla-central
https://treeherder.mozilla.org/logviewer.html#?job_id=2929818&repo=mozilla-central
https://treeherder.mozilla.org/logviewer.html#?job_id=2928800&repo=mozilla-central
https://treeherder.mozilla.org/logviewer.html#?job_id=2929649&repo=mozilla-central
(Reporter)

Updated

2 years ago
Keywords: intermittent-failure

Comment 1

2 years ago
31 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 10
* fx-team: 8
* try: 6
* b2g-inbound: 6
* mozilla-central: 1

Platform breakdown:
* b2g-device-image: 21
* b2g-linux64: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1234416&startday=2015-12-22&endday=2015-12-22&tree=all
this seems to have come out of the blue- the code for poolmanager was added on Nov 30 from:
https://hg.mozilla.org/mozilla-central/rev/d0e882fcbbf7

Now 3 weeks later we are having issues?

:catlee, you had reviewed the code that added the reference to pool manager, is there a reason why this package isn't available for these jobs?
Flags: needinfo?(catlee)
catlee is on pto, maybe coop would have another lead
Flags: needinfo?(catlee) → needinfo?(coop)

Comment 4

2 years ago
21 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 7
* b2g-inbound: 7
* mozilla-central: 4
* fx-team: 2
* mozilla-b2g44_v2_5: 1

Platform breakdown:
* b2g-device-image: 15
* b2g-linux64: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1234416&startday=2015-12-23&endday=2015-12-23&tree=all

Updated

2 years ago
Depends on: 1207688

Comment 5

2 years ago
I've reopened the bug 1207688. The python + openssl hackery there seems to have been more fragile than expected.
Flags: needinfo?(coop)

Comment 6

2 years ago
102 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 32
* b2g-inbound: 27
* fx-team: 17
* mozilla-central: 16
* try: 8
* mozilla-b2g44_v2_5: 2

Platform breakdown:
* b2g-device-image: 70
* b2g-linux64: 31
* b2g-emu-l: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1234416&startday=2015-12-21&endday=2015-12-27&tree=all
The traceback about being unable to import requests is a warning -- all that code is intended to be optional and shouldn't impact the rest of the job. The errors related to git later in the log look more relevant.
No longer depends on: 1207688

Comment 8

2 years ago
15 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 10
* b2g-inbound: 3
* fx-team: 2

Platform breakdown:
* b2g-device-image: 12
* b2g-emu-ics: 2
* b2g-linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1234416&startday=2015-12-28&endday=2015-12-28&tree=all

Comment 9

2 years ago
21 automation job failures were associated with this bug yesterday.

Repository breakdown:
* b2g-inbound: 8
* mozilla-inbound: 6
* mozilla-central: 4
* fx-team: 2
* try: 1

Platform breakdown:
* b2g-device-image: 16
* b2g-linux64: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1234416&startday=2015-12-29&endday=2015-12-29&tree=all

Comment 10

2 years ago
23 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 11
* b2g-inbound: 7
* fx-team: 4
* try: 1

Platform breakdown:
* b2g-device-image: 17
* b2g-linux64: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1234416&startday=2015-12-31&endday=2015-12-31&tree=all

Comment 11

2 years ago
107 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 32
* b2g-inbound: 29
* mozilla-central: 23
* fx-team: 16
* try: 7

Platform breakdown:
* b2g-device-image: 80
* b2g-linux64: 25
* b2g-emu-ics: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1234416&startday=2015-12-28&endday=2016-01-03&tree=all

Comment 12

2 years ago
29 automation job failures were associated with this bug yesterday.

Repository breakdown:
* fx-team: 13
* b2g-inbound: 6
* mozilla-inbound: 2
* mozilla-b2g44_v2_5: 2
* mozilla-b2g37_v2_2r: 2
* mozilla-b2g37_v2_2: 2
* try: 1
* mozilla-central: 1

Platform breakdown:
* b2g-device-image: 23
* b2g-linux64: 3
* b2g-emu-l: 2
* b2g-emu-jb: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1234416&startday=2016-01-04&endday=2016-01-04&tree=all
Created attachment 8704362 [details]
MozReview Request: Bug 1234416 - Check that userconfig exists before trying to remove. r=jlund

Review commit: https://reviewboard.mozilla.org/r/29647/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/29647/
Attachment #8704362 - Flags: review?(jlund)
Yeah, the poolmanager change isn't related, nor the python+openssl hacks. This looks like the normal 'repo' suckage. Something is causing 'HEAD' in some of the git repos to not exist or be invalid.

We have some minor cleanup to do in the post-run handling to remove the spurious exception there.

Comment 15

2 years ago
87 automation job failures were associated with this bug yesterday.

Repository breakdown:
* b2g-inbound: 23
* mozilla-inbound: 22
* fx-team: 20
* mozilla-central: 12
* mozilla-b2g44_v2_5: 7
* try: 2
* mozilla-b2g37_v2_2: 1

Platform breakdown:
* b2g-device-image: 68
* b2g-emu-ics: 11
* b2g-linux64: 5
* b2g-emu-l: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1234416&startday=2016-01-05&endday=2016-01-05&tree=all
Comment on attachment 8704362 [details]
MozReview Request: Bug 1234416 - Check that userconfig exists before trying to remove. r=jlund

https://reviewboard.mozilla.org/r/29647/#review26499

lgtm :)

::: testing/mozharness/scripts/b2g_build.py:1150
(Diff revision 1)
> +        if os.path.exists(userconfig_path):

another option is to use self.rmtree

https://dxr.mozilla.org/mozilla-central/rev/29258f59e5456a1a518ccce6b473b50c1173477e/testing/mozharness/mozharness/base/script.py#162

this will check for path existance for you, then, if it's not a dir, it will use os.remove but throw in some mh logging and retry logic to boot
Attachment #8704362 - Flags: review?(jlund) → review+

Comment 17

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/18fedb75f8b0

Comment 18

2 years ago
108 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 36
* b2g-inbound: 26
* fx-team: 25
* mozilla-central: 20
* try: 1

Platform breakdown:
* b2g-device-image: 107
* b2g-linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1234416&startday=2016-01-06&endday=2016-01-06&tree=all

Comment 19

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/18fedb75f8b0
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox46: --- → fixed
Resolution: --- → FIXED
we've updated the git tarballs used by automation - hopefully that helps this issue.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Comment 21

2 years ago
31 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 8
* fx-team: 8
* b2g-inbound: 8
* mozilla-central: 5
* try: 2

Platform breakdown:
* b2g-device-image: 29
* b2g-linux64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1234416&startday=2016-01-07&endday=2016-01-07&tree=all

Comment 22

2 years ago
269 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 69
* fx-team: 67
* b2g-inbound: 66
* mozilla-central: 40
* try: 12
* mozilla-b2g44_v2_5: 9
* mozilla-b2g37_v2_2: 4
* mozilla-b2g37_v2_2r: 2

Platform breakdown:
* b2g-device-image: 234
* b2g-linux64: 18
* b2g-emu-ics: 11
* b2g-emu-l: 5
* b2g-emu-jb: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1234416&startday=2016-01-04&endday=2016-01-10&tree=all

Comment 23

2 years ago
5 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* try: 5

Platform breakdown:
* b2g-linux64: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1234416&startday=2016-01-11&endday=2016-01-17&tree=all
(Reporter)

Updated

a year ago
Keywords: intermittent-failure

Updated

9 months ago
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago9 months ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.