bugzilla.mozilla.org has resumed normal operation. Attachments prior to 2014 will be unavailable for a few days. This is tracked in Bug 1475801.
Please report any other irregularities here.

Intermittent [taskcluster:error] Task timeout after 14400 seconds. Force killing container.

RESOLVED WONTFIX

Status

Taskcluster
General
RESOLVED WONTFIX
3 years ago
3 years ago

People

(Reporter: RyanVM, Unassigned)

Tracking

({intermittent-failure})

Details

Comment hidden (empty)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Summary: Intermittent [taskcluster] Error: Task timeout after 14400 seconds. Force killing container. → Intermittent [taskcluster:error] Task timeout after 14400 seconds. Force killing container.
Duplicate of this bug: 1204235
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)

Comment 57

3 years ago
11 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 7
* b2g-inbound: 3
* fx-team: 1

Platform breakdown:
* b2g-emu-jb: 5
* b2g-device-image: 4
* b2g-emu-l: 2

For more details, see:
http://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-09-28&endday=2015-10-04&tree=all

Comment 58

3 years ago
25 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 13
* b2g-inbound: 9
* fx-team: 3

Platform breakdown:
* b2g-emu-l: 8
* b2g-device-image: 7
* b2g-emu-ics: 5
* b2g-emu-jb: 3
* b2g-emu-kk: 2

For more details, see:
http://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-10-07&endday=2015-10-07&tree=all

Comment 59

3 years ago
27 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 12
* b2g-inbound: 10
* fx-team: 5

Platform breakdown:
* b2g-device-image: 12
* b2g-emu-kk: 5
* b2g-emu-x86-kk: 4
* b2g-emu-jb: 4
* b2g-emu-l: 2

For more details, see:
http://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-10-08&endday=2015-10-08&tree=all

Comment 60

3 years ago
99 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 44
* fx-team: 27
* b2g-inbound: 26
* mozilla-central: 2

Platform breakdown:
* b2g-device-image: 34
* b2g-emu-l: 21
* b2g-emu-kk: 14
* b2g-emu-jb: 14
* b2g-emu-x86-kk: 8
* b2g-emu-ics: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-10-05&endday=2015-10-11&tree=all

Comment 61

3 years ago
18 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 14
* mozilla-central: 3
* fx-team: 1

Platform breakdown:
* b2g-emu-l: 4
* b2g-device-image: 4
* b2g-emu-x86-kk: 3
* b2g-emu-kk: 3
* b2g-emu-jb: 3
* b2g-emu-ics: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-10-12&endday=2015-10-12&tree=all

Comment 62

3 years ago
23 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 11
* b2g-inbound: 7
* fx-team: 3
* mozilla-central: 2

Platform breakdown:
* b2g-emu-jb: 6
* b2g-device-image: 5
* b2g-emu-ics: 4
* b2g-emu-l: 3
* b2g-emu-kk: 3
* b2g-emu-x86-kk: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-10-14&endday=2015-10-14&tree=all

Comment 63

3 years ago
29 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 9
* fx-team: 9
* mozilla-central: 5
* b2g-inbound: 5
* try: 1

Platform breakdown:
* b2g-device-image: 8
* b2g-emu-x86-kk: 7
* b2g-emu-jb: 6
* b2g-emu-kk: 4
* b2g-emu-l: 3
* b2g-emu-ics: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-10-15&endday=2015-10-15&tree=all

Comment 64

3 years ago
30 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 17
* b2g-inbound: 7
* fx-team: 5
* mozilla-central: 1

Platform breakdown:
* b2g-emu-kk: 9
* b2g-emu-jb: 6
* b2g-emu-l: 5
* b2g-emu-x86-kk: 4
* b2g-device-image: 4
* b2g-emu-ics: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-10-16&endday=2015-10-16&tree=all

Comment 65

3 years ago
134 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 65
* b2g-inbound: 29
* fx-team: 24
* mozilla-central: 14
* try: 2

Platform breakdown:
* b2g-emu-jb: 33
* b2g-device-image: 26
* b2g-emu-kk: 25
* b2g-emu-x86-kk: 22
* b2g-emu-l: 17
* b2g-emu-ics: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-10-12&endday=2015-10-18&tree=all

Comment 66

3 years ago
28 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 15
* b2g-inbound: 10
* fx-team: 2
* try: 1

Platform breakdown:
* b2g-device-image: 11
* b2g-emu-l: 5
* b2g-emu-ics: 4
* b2g-emu-kk: 3
* b2g-emu-jb: 3
* b2g-emu-x86-kk: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-10-21&endday=2015-10-21&tree=all

Comment 67

3 years ago
88 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 39
* b2g-inbound: 36
* fx-team: 12
* try: 1

Platform breakdown:
* b2g-device-image: 30
* b2g-emu-ics: 15
* b2g-emu-l: 12
* b2g-emu-kk: 11
* b2g-emu-x86-kk: 10
* b2g-emu-jb: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-10-19&endday=2015-10-25&tree=all

Comment 68

3 years ago
15 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 7
* fx-team: 4
* b2g-inbound: 3
* mozilla-central: 1

Platform breakdown:
* b2g-device-image: 7
* b2g-emu-kk: 3
* b2g-emu-l: 2
* b2g-emu-jb: 2
* b2g-emu-x86-kk: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-10-26&endday=2015-10-26&tree=all

Comment 69

3 years ago
21 automation job failures were associated with this bug yesterday.

Repository breakdown:
* fx-team: 8
* b2g-inbound: 7
* mozilla-inbound: 6

Platform breakdown:
* b2g-emu-ics: 5
* b2g-emu-l: 4
* b2g-emu-x86-kk: 3
* b2g-emu-kk: 3
* b2g-emu-jb: 3
* b2g-device-image: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-10-27&endday=2015-10-27&tree=all

Comment 70

3 years ago
17 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 6
* fx-team: 5
* mozilla-central: 3
* b2g-inbound: 3

Platform breakdown:
* b2g-emu-kk: 5
* b2g-emu-l: 4
* b2g-emu-ics: 4
* b2g-device-image: 2
* b2g-emu-x86-kk: 1
* b2g-emu-jb: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-10-29&endday=2015-10-29&tree=all

Comment 71

3 years ago
89 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 35
* fx-team: 32
* b2g-inbound: 16
* mozilla-central: 6

Platform breakdown:
* b2g-emu-ics: 20
* b2g-emu-l: 16
* b2g-emu-kk: 16
* b2g-device-image: 16
* b2g-emu-jb: 13
* b2g-emu-x86-kk: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-10-26&endday=2015-11-01&tree=all

Comment 72

3 years ago
16 automation job failures were associated with this bug yesterday.

Repository breakdown:
* b2g-inbound: 7
* mozilla-inbound: 6
* mozilla-central: 2
* fx-team: 1

Platform breakdown:
* b2g-emu-l: 4
* b2g-emu-ics: 4
* b2g-emu-x86-kk: 3
* b2g-emu-jb: 3
* b2g-emu-kk: 1
* b2g-device-image: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-11-04&endday=2015-11-04&tree=all

Comment 73

3 years ago
20 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 14
* fx-team: 5
* mozilla-b2g44_v2_5: 1

Platform breakdown:
* b2g-emu-l: 6
* b2g-emu-ics: 4
* b2g-emu-x86-kk: 3
* b2g-device-image: 3
* b2g-emu-kk: 2
* b2g-emu-jb: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-11-05&endday=2015-11-05&tree=all

Comment 74

3 years ago
25 automation job failures were associated with this bug yesterday.

Repository breakdown:
* fx-team: 8
* b2g-inbound: 8
* mozilla-inbound: 7
* mozilla-central: 2

Platform breakdown:
* b2g-emu-l: 5
* b2g-emu-kk: 5
* b2g-emu-jb: 5
* b2g-device-image: 4
* b2g-emu-x86-kk: 3
* b2g-emu-ics: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-11-06&endday=2015-11-06&tree=all

Comment 75

3 years ago
134 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 64
* b2g-inbound: 37
* fx-team: 25
* mozilla-central: 7
* mozilla-b2g44_v2_5: 1

Platform breakdown:
* b2g-emu-l: 32
* b2g-device-image: 26
* b2g-emu-kk: 25
* b2g-emu-x86-kk: 22
* b2g-emu-ics: 15
* b2g-emu-jb: 14

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-11-02&endday=2015-11-08&tree=all

Comment 76

3 years ago
15 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 11
* mozilla-b2g44_v2_5: 2
* fx-team: 1
* b2g-inbound: 1

Platform breakdown:
* b2g-emu-kk: 6
* b2g-device-image: 3
* b2g-emu-x86-kk: 2
* b2g-emu-l: 2
* b2g-emu-jb: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-11-09&endday=2015-11-09&tree=all

Comment 77

3 years ago
46 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 25
* fx-team: 10
* b2g-inbound: 9
* mozilla-b2g44_v2_5: 2

Platform breakdown:
* b2g-device-image: 13
* b2g-emu-kk: 10
* b2g-emu-l: 8
* b2g-emu-jb: 7
* b2g-emu-x86-kk: 6
* b2g-emu-ics: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-11-09&endday=2015-11-15&tree=all

Comment 78

3 years ago
20 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 13
* b2g-inbound: 5
* fx-team: 2

Platform breakdown:
* b2g-emu-kk: 5
* b2g-device-image: 5
* b2g-emu-x86-kk: 3
* b2g-emu-l: 3
* b2g-emu-jb: 2
* b2g-emu-ics: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-11-20&endday=2015-11-20&tree=all

Comment 79

3 years ago
70 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 52
* b2g-inbound: 9
* fx-team: 7
* mozilla-central: 2

Platform breakdown:
* b2g-emu-l: 16
* b2g-emu-kk: 14
* b2g-emu-jb: 13
* b2g-device-image: 12
* b2g-emu-x86-kk: 10
* b2g-emu-ics: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-11-16&endday=2015-11-22&tree=all

Comment 80

3 years ago
63 automation job failures were associated with this bug yesterday.

Repository breakdown:
* fx-team: 28
* mozilla-inbound: 25
* b2g-inbound: 8
* try: 2

Platform breakdown:
* b2g-device-image: 37
* b2g-emu-l: 7
* b2g-emu-kk: 7
* b2g-emu-jb: 5
* b2g-emu-x86-kk: 4
* windows8-64: 1
* windows7-32: 1
* b2g-emu-ics: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-11-26&endday=2015-11-26&tree=all

Comment 81

3 years ago
99 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 41
* fx-team: 31
* b2g-inbound: 24
* try: 2
* mozilla-central: 1

Platform breakdown:
* b2g-device-image: 45
* b2g-emu-kk: 18
* b2g-emu-l: 13
* b2g-emu-jb: 11
* b2g-emu-x86-kk: 5
* b2g-emu-ics: 5
* windows8-64: 1
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-11-23&endday=2015-11-29&tree=all

Comment 82

3 years ago
18 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 10
* b2g-inbound: 6
* fx-team: 2

Platform breakdown:
* b2g-emu-jb: 5
* b2g-emu-ics: 5
* b2g-emu-x86-kk: 3
* b2g-emu-l: 2
* b2g-device-image: 2
* b2g-emu-kk: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-12-01&endday=2015-12-01&tree=all

Comment 83

3 years ago
20 automation job failures were associated with this bug yesterday.

Repository breakdown:
* b2g-inbound: 8
* mozilla-inbound: 7
* fx-team: 5

Platform breakdown:
* b2g-emu-l: 6
* b2g-device-image: 6
* b2g-emu-kk: 4
* b2g-emu-ics: 2
* b2g-emu-x86-kk: 1
* b2g-emu-jb: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-12-02&endday=2015-12-02&tree=all

Comment 84

3 years ago
23 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 12
* b2g-inbound: 7
* fx-team: 4

Platform breakdown:
* b2g-emu-jb: 7
* b2g-device-image: 5
* b2g-emu-l: 3
* b2g-emu-kk: 3
* b2g-emu-ics: 3
* b2g-emu-x86-kk: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-12-04&endday=2015-12-04&tree=all

Comment 85

3 years ago
105 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 55
* b2g-inbound: 27
* fx-team: 20
* mozilla-central: 3

Platform breakdown:
* b2g-emu-jb: 28
* b2g-device-image: 22
* b2g-emu-ics: 17
* b2g-emu-kk: 15
* b2g-emu-l: 12
* b2g-emu-x86-kk: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-11-30&endday=2015-12-06&tree=all

Comment 86

3 years ago
23 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 10
* fx-team: 8
* b2g-inbound: 4
* mozilla-central: 1

Platform breakdown:
* b2g-emu-x86-kk: 6
* b2g-emu-l: 5
* b2g-emu-jb: 4
* b2g-emu-ics: 4
* b2g-device-image: 3
* b2g-emu-kk: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-12-07&endday=2015-12-13&tree=all

Comment 87

3 years ago
16 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 10
* b2g-inbound: 5
* fx-team: 1

Platform breakdown:
* b2g-emu-kk: 4
* b2g-device-image: 4
* b2g-emu-ics: 3
* b2g-emu-x86-kk: 2
* b2g-emu-jb: 2
* b2g-emu-l: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-12-14&endday=2015-12-14&tree=all

Comment 88

3 years ago
21 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 9
* fx-team: 6
* b2g-inbound: 6

Platform breakdown:
* b2g-device-image: 5
* b2g-emu-kk: 4
* b2g-emu-ics: 4
* b2g-emu-x86-kk: 3
* b2g-emu-jb: 3
* b2g-emu-l: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-12-16&endday=2015-12-16&tree=all

Comment 89

3 years ago
17 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 8
* b2g-inbound: 7
* mozilla-central: 1
* fx-team: 1

Platform breakdown:
* b2g-device-image: 5
* b2g-emu-kk: 4
* b2g-emu-jb: 4
* b2g-emu-x86-kk: 2
* b2g-emu-l: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-12-17&endday=2015-12-17&tree=all

Comment 90

3 years ago
27 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 17
* fx-team: 4
* b2g-inbound: 4
* mozilla-central: 2

Platform breakdown:
* b2g-emu-kk: 8
* b2g-emu-l: 6
* b2g-device-image: 6
* b2g-emu-ics: 4
* b2g-emu-jb: 2
* b2g-emu-x86-kk: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-12-18&endday=2015-12-18&tree=all

Comment 91

3 years ago
101 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 60
* b2g-inbound: 25
* fx-team: 13
* mozilla-central: 3

Platform breakdown:
* b2g-emu-kk: 26
* b2g-device-image: 24
* b2g-emu-jb: 15
* b2g-emu-ics: 14
* b2g-emu-l: 12
* b2g-emu-x86-kk: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-12-14&endday=2015-12-20&tree=all
This is the #6 intermittent orange over the past 3 days.  It would be helpful to get this assigned appropriately (whether that's a fix in Taskcluster or better diagnostics that would lead to a fix elsewhere).
Flags: needinfo?(sdeckelmann)

Comment 93

3 years ago
17 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 9
* b2g-inbound: 5
* fx-team: 2
* mozilla-central: 1

Platform breakdown:
* b2g-emu-jb: 5
* b2g-emu-ics: 4
* b2g-emu-x86-kk: 3
* b2g-emu-l: 3
* b2g-emu-kk: 1
* b2g-device-image: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-12-22&endday=2015-12-22&tree=all
ack, these failures are all git sync related, some examples:
-------------------------------------------------------------------------------
 Fetching project android-development

 Fetching project platform/external/liblzf

 Fetching project platform/external/iproute2

 Fetching project platform/abi/cpp

 Fetching project platform/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.7-4.6

 [taskcluster-vcs:warning] run end (with error) try (10/20) retrying in 11629572.478532791 ms : ./repo sync -j1

 [taskcluster:error] Task timeout after 14400 seconds. Force killing container.

-------------------------------------------------------------------------------
^ this is the most common one


-------------------------------------------------------------------------------
 : export GIT_DIR=/home/worker/workspace/B2G/.repo/projects/external/icu4c.git

 : git fetch caf --tags +refs/heads/*:refs/remotes/caf/* 1>| 2>|

 : load refs /home/worker/workspace/B2G/.repo/projects/external/icu4c.git

 : scan refs /home/worker/workspace/B2G/.repo/projects/external/icu4c.git

 : parsing /home/worker/workspace/B2G/.repo/projects/external/aac.git/config

 

 : export GIT_DIR=/home/worker/workspace/B2G/.repo/projects/external/aac.git

 : git fetch caf --tags +refs/heads/*:refs/remotes/caf/* 1>| 2>|

 [taskcluster:error] Task timeout after 14400 seconds. Force killing container.
-------------------------------------------------------------------------------
^ this is not as common, but seen many times


-------------------------------------------------------------------------------
  * [new tag]         android-6.0.0_r7 -> android-6.0.0_r7

  * [new tag]         android-6.0.1_r1 -> android-6.0.1_r1

  * [new tag]         android-6.0.1_r3 -> android-6.0.1_r3

  * [new tag]         android-cts-6.0_r2 -> android-cts-6.0_r2

 : load refs /home/worker/workspace/B2G/.repo/projects/external/expat.git

 : scan refs /home/worker/workspace/B2G/.repo/projects/external/expat.git

 : git update-ref -m manifest set to 4a65b6cb6d2aae6bf952b0e6b44881f84fdf5128 --no-deref refs/remotes/m/master 4a65b6cb6d2aae6bf952b0e6b44881f84fdf5128^0 1>| 2>|

 : parsing /home/worker/workspace/B2G/.repo/projects/external/strace.git/config

 

 : export GIT_DIR=/home/worker/workspace/B2G/.repo/projects/external/strace.git

 : git fetch caf --tags +refs/heads/*:refs/remotes/caf/* 1>| 2>|

 [taskcluster:error] Task timeout after 14400 seconds. Force killing container.
-------------------------------------------------------------------------------
^ this is seen many times as well


:gps, you do a lot (at least blog about) vcs, can you help find the right person to look into why our git fetch is taking so long?
Flags: needinfo?(gps)
Git + repo have historically been a cluster$*#@.

One reason things frequently time out is because we're fetching an insane amount of data and repositories. There's a lot of surface area for things to go wrong.

On top of that, repo can be brain dead about retries. My understanding is it will aggressively purge old repos and resync from scratch. This has led to traffic floods against git.mo in the past, which makes clones fail, which makes jobs fail.

hwine, fubar, and/or catlee should be able to add more context, as they have all touched the git/repo side of things more than me.
Flags: needinfo?(gps)

Comment 96

3 years ago
58 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 33
* b2g-inbound: 13
* fx-team: 11
* mozilla-central: 1

Platform breakdown:
* b2g-emu-jb: 15
* b2g-emu-ics: 14
* b2g-emu-l: 12
* b2g-device-image: 8
* b2g-emu-kk: 5
* b2g-emu-x86-kk: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-12-21&endday=2015-12-27&tree=all
(In reply to Joel Maher (:jmaher) from comment #94)
>
>  Fetching project
> platform/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.7-4.6
> 
>  [taskcluster-vcs:warning] run end (with error) try (10/20) retrying in
> 11629572.478532791 ms : ./repo sync -j1
> 
>  [taskcluster:error] Task timeout after 14400 seconds. Force killing
> container.

offhand, the only odd thing I see is that i686-linux-glibc2.7-4.6.git, aac.git, and strace.git are
missing the remote default branch:

Cloning into 'i686-linux-glibc2.7-4.6'...
[...].
warning: remote HEAD refers to nonexistent ref, unable to checkout.

also:
git1.dmz.scl3# pwd
/var/lib/gitolite3/repositories/external/sprd-aosp/platform/external/strace.git
git1.dmz.scl3# git fsck
notice: HEAD points to an unborn branch (master)
Checking object directories: 100% (256/256), done.
Checking objects: 100% (343/343), done.

but that's probably a red herring? 

the other thing to look at is to make sure that TC is actually caching those repos.
we need to figure out a way to fail faster here, 3 hours and we are stuck on vcs cloning sounds horrible.

Either way, who should own this?  if gps says git+repo has a bad history, why not use a better vcs?  I know that isn't a conversation for this bug, but we have our top failures wasting every single developers time related to git, even if they don't use it.  

I know this is the holidays, but we shouldn't have to wait for Selena to get back in order to assign this to someone to fix this issue, unless Selena is the one who owns/wrote/maintains that part of the code.

catlee, can you find an owner and get this fixed?  (it appears hwine is on pto this week as well)
Flags: needinfo?(catlee)

Comment 99

3 years ago
17 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* b2g-inbound: 8
* mozilla-inbound: 3
* mozilla-central: 3
* fx-team: 3

Platform breakdown:
* b2g-device-image: 5
* b2g-emu-jb: 4
* b2g-emu-l: 3
* b2g-emu-kk: 3
* b2g-emu-x86-kk: 1
* b2g-emu-ics: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2015-12-28&endday=2016-01-03&tree=all
I'm not familiar with how tc-vcs handles this. For buildbot, we've done some work to optimize git+repo usage, but I'm not sure if tc-vcs is doing the same thing.

Greg / Jonas / John - any suggestions here?
Flags: needinfo?(jopsen)
Flags: needinfo?(jhford)
Flags: needinfo?(garndt)
Flags: needinfo?(catlee)
The last time I looked into this I hit a dead end and came to the same conclusion that for some reason 'repo' was doing full clones of all the projects, even if there was a local copy available.  It wasn't clear to me at the time why it was being forced to do such a thing.

I spot checked a few of the recently starred tasks and there are some that already had a locally cached copy that tc-vcs pulled for a previous task, and some where it had to do a fresh checkout.  Both types of tasks resulted in the same outcome...timing out when running 'repo sync -j1'.

What were the things done to optimize git+repo on the buildbot side? Perhaps there are things we should be doing as well that could help.
Flags: needinfo?(garndt)
The code for this is here:
https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/building/buildb2gbase.py#451

I don't think we run with -j1, so that may be a quick optimization.
We symlink the local working directory's .repo dir to a machine-local cache.
In the case of some of these tasks that timed out, .repo is already populated with a local directory cache from a previous task.  We attempt to do that as well after a task has already pulled it down the first time. 

re: -j1 
We originally had this at -j100 which was way too high and we went in the other drastic direction of making it -j1 because of the problems it was causing with git.m.o repeatedly. When repo gets into a state of wanting to do full clones, which happens on multiple machines around the same time, it creates quite the situation with git.m.o.  Since changing this to -j1 I have not had a report of taskcluster tasks causing major disruptions of git.m.o so I'm a little scared to change that, but if there is a more sensible option I'm always up for that. :)
Flags: needinfo?(jopsen)

Comment 104

3 years ago
40 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 14
* mozilla-central: 11
* fx-team: 9
* b2g-inbound: 6

Platform breakdown:
* b2g-device-image: 12
* b2g-emu-jb: 7
* b2g-emu-l: 6
* b2g-emu-kk: 6
* b2g-emu-ics: 5
* b2g-emu-x86-kk: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2016-01-04&endday=2016-01-10&tree=all

Comment 105

3 years ago
30 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 18
* fx-team: 6
* b2g-inbound: 4
* mozilla-central: 2

Platform breakdown:
* b2g-emu-ics: 8
* b2g-emu-jb: 6
* b2g-device-image: 6
* b2g-emu-l: 4
* b2g-emu-x86-kk: 3
* b2g-emu-kk: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2016-01-14&endday=2016-01-14&tree=all

Comment 106

3 years ago
78 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 50
* b2g-inbound: 12
* fx-team: 10
* mozilla-central: 6

Platform breakdown:
* b2g-emu-l: 16
* b2g-emu-jb: 16
* b2g-device-image: 15
* b2g-emu-ics: 13
* b2g-emu-kk: 10
* b2g-emu-x86-kk: 7
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2016-01-11&endday=2016-01-17&tree=all

Comment 107

3 years ago
34 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 24
* fx-team: 5
* mozilla-central: 4
* try: 1

Platform breakdown:
* b2g-emu-ics: 9
* b2g-emu-x86-kk: 7
* b2g-device-image: 7
* b2g-emu-kk: 5
* b2g-emu-jb: 4
* b2g-emu-l: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1198092&startday=2016-01-18&endday=2016-01-24&tree=all
Given that these primarily affect b2g emulator builds, I'm closing this bug. 

We have work underway to replace tc-vcs that will probably also fix issues involving repo, but this is no longer something our team will invest time in fixing.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Flags: needinfo?(sdeckelmann)
Flags: needinfo?(jhford)
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.