1392349 - Intermittent upload-symbols [taskcluster:error] Task timeout after 600 seconds. Force killing container.

Status: RESOLVED FIXED

(Taskcluster :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: rvandermeulen [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=124613735&repo=mozilla-beta

https://queue.taskcluster.net/v1/task/T_dU9DkSQOqjMDLpoHme3Q/runs/0/artifacts/public/logs/live_backing.log

Comment 1

[(not currently active) Ted Mielczarek]

From that log:
+ wget https://queue.taskcluster.net/v1/task/YC0FgOlETmS8P6H4wnKTCg/artifacts/public/build/target.crashreporter-symbols-full.zip
--2017-08-21 17:23:42--  https://queue.taskcluster.net/v1/task/YC0FgOlETmS8P6H4wnKTCg/artifacts/public/build/target.crashreporter-symbols-full.zip

<...>

HTTP request sent, awaiting response... 200 OK
Length: 1128789115 (1.1G) [application/zip]

<...>

2017-08-21 17:28:36 (4.06 MB/s) - 'target.crashreporter-symbols-full.zip' saved [1128789115/1128789115]

<...>

[taskcluster:error] Task timeout after 600 seconds. Force killing container.
[taskcluster 2017-08-21 17:33:41.204Z] === Task Finished ===

The symbols zip file is 1.1G, and it takes over 4 minutes just to download it, and then it has to upload it again! We might just need to increase the symbol upload task's time limit.

Comment 2

[(not currently active) Ted Mielczarek]

Also I filed bug 1392400 on a potential improvement here--we ought to be able to just give the symbol api the URL to the symbols.zip from the task and let it download the zip itself, so that we don't have to download and then re-upload the whole thing.

Comment 3

[Intermittent Failures Robot]

6 failures in 908 pushes (0.007 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 5
* mozilla-central: 1

Platform breakdown:
* linux64: 3
* linux32: 2
* android-5-0-aarch64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1392349&startday=2017-08-21&endday=2017-08-27&tree=all

Comment 4

[Intermittent Failures Robot]

7 failures in 939 pushes (0.007 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 7

Platform breakdown:
* linux32: 5
* linux64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1392349&startday=2017-08-28&endday=2017-09-03&tree=all

Comment 5

[Intermittent Failures Robot]

9 failures in 924 pushes (0.01 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 7
* mozilla-central: 2

Platform breakdown:
* linux32: 5
* linux64: 2
* windows2012-32: 1
* android-5-0-aarch64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1392349&startday=2017-09-04&endday=2017-09-10&tree=all

Comment 6

[Intermittent Failures Robot]

14 failures in 1032 pushes (0.014 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 13
* mozilla-central: 1

Platform breakdown:
* linux64: 8
* linux32: 5
* android-4-2-x86-old-id: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1392349&startday=2017-09-11&endday=2017-09-17&tree=all

Comment 7

[Intermittent Failures Robot]

2 failures in 943 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 2

Platform breakdown:
* linux64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1392349&startday=2017-09-18&endday=2017-09-24&tree=all

Comment 8

[Intermittent Failures Robot]

1 failures in 885 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 1

Platform breakdown:
* linux64-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1392349&startday=2017-09-25&endday=2017-10-01&tree=all

Comment 9

[Intermittent Failures Robot]

4 failures in 864 pushes (0.005 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 4

Platform breakdown:
* windows2012-32-devedition: 2
* windows2012-64-devedition: 1
* windows2012-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1392349&startday=2017-10-16&endday=2017-10-22&tree=all

Comment 10

[Intermittent Failures Robot]

2 failures in 912 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 2

Platform breakdown:
* linux64-devedition: 1
* android-5-0-aarch64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1392349&startday=2017-10-23&endday=2017-10-29&tree=all

Comment 11

[Intermittent Failures Robot]

9 failures in 857 pushes (0.011 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 8
* mozilla-central: 1

Platform breakdown:
* linux32: 4
* windows2012-32-devedition: 1
* linux64-devedition: 1
* linux32-devedition: 1
* android-5-0-aarch64: 1
* android-4-0-armv7-api16-old-id: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1392349&startday=2017-10-30&endday=2017-11-05&tree=all

Comment 12

[Intermittent Failures Robot]

7 failures in 849 pushes (0.008 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-release: 5
* mozilla-beta: 2

Platform breakdown:
* linux32-devedition: 4
* osx-cross: 1
* linux32: 1
* android-4-0-armv7-api16-old-id: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1392349&startday=2017-11-06&endday=2017-11-12&tree=all

Comment 13

[Intermittent Failures Robot]

2 failures in 762 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 2

Platform breakdown:
* linux64: 1
* linux32-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1392349&startday=2017-11-13&endday=2017-11-19&tree=all

Comment 14

[Intermittent Failures Robot]

6 failures in 744 pushes (0.008 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 3
* mozilla-release: 2
* mozilla-central: 1

Platform breakdown:
* linux64: 3
* linux32: 2
* linux32-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1392349&startday=2017-11-20&endday=2017-11-26&tree=all

Comment 15

[Intermittent Failures Robot]

16 failures in 155 pushes (0.103 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-beta: 13
* mozilla-release: 2
* mozilla-central: 1

Platform breakdown:
* linux64-devedition: 4
* linux32-devedition: 3
* linux32: 3
* windows2012-64-devedition: 2
* windows2012-64: 1
* windows2012-32-devedition: 1
* windows2012-32: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1392349&startday=2017-11-29&endday=2017-11-29&tree=all

Comment 16

[Intermittent Failures Robot]

33 failures in 792 pushes (0.042 failures/push) were associated with this bug in the last 7 days. 

This is the #29 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-beta: 20
* mozilla-release: 9
* mozilla-central: 4

Platform breakdown:
* linux32: 7
* linux64-devedition: 5
* linux32-devedition: 5
* windows2012-64-devedition: 3
* windows2012-64: 3
* linux64: 3
* windows2012-32: 2
* android-5-0-aarch64: 2
* windows2012-32-devedition: 1
* android-4-2-x86: 1
* android-4-0-armv7-api16: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1392349&startday=2017-11-27&endday=2017-12-03&tree=all

Comment 17

[Intermittent Failures Robot]

6 failures in 889 pushes (0.007 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 4
* mozilla-release: 1
* mozilla-central: 1

Platform breakdown:
* linux32-devedition: 3
* osx-cross: 1
* linux64-devedition: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1392349&startday=2017-12-04&endday=2017-12-10&tree=all

Comment 18

[(not currently active) Ted Mielczarek]

I bumped the max run time of the symbol-upload task to 900 seconds in bug 1422740, and I just autolanded bug 1423881 which should make the task run much faster (since it no longer has to download the symbols zip locally), so I'm going to preemptively mark this as FIXED.

Interestingly, looking at the orangefactor link in comment 17, a few of those are upload-generated-sources tasks, which is a bummer.

Comment 19

[Intermittent Failures Robot]

1 failures in 462 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-release: 1

Platform breakdown:
* osx-cross-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1392349&startday=2018-01-01&endday=2018-01-07&tree=all

Comment 20

[Intermittent Failures Robot]

3 failures in 788 pushes (0.004 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 3

Platform breakdown:
* windows2012-64-devedition: 2
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1392349&startday=2018-01-08&endday=2018-01-14&tree=all

Comment 21

[Intermittent Failures Robot]

5 failures in 701 pushes (0.007 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-release: 5

Platform breakdown:
* linux64-devedition: 4
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1392349&startday=2018-01-22&endday=2018-01-28&tree=all