Linux/Android Symbol upload failed

NEW
Unassigned

Status

Release Engineering
General
4 months ago
4 months ago

People

(Reporter: Tomcat, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(URL)

(Reporter)

Description

4 months ago
like https://public-artifacts.taskcluster.net/IrvP6mJgTv6RFe9m9hxgxA/0/public/logs/live_backing.log 

Uploading symbol file "target.crashreporter-symbols-full.zip" to "https://crash-stats.mozilla.com/symbols/upload"
Attempt 1 of 5...

[taskcluster:error] Task timeout after 600 seconds. Force killing container.
[taskcluster 2017-07-07 11:06:29.834Z] === Task Finished ===
[taskcluster 2017-07-07 11:06:29.834Z] Unsuccessful task run with exit code: -1 completed in 603.179 seconds

not sure why 1/5 and what happened to 2/3/4/5 but we should investigate
(Reporter)

Comment 1

4 months ago
also hit linux on beta
Summary: Android Symbol upload failed → Linux/Android Symbol upload failed
FWIW, this is the error we get on crash-stats: 

> UnreadablePostError
> timeout during read(65536) on wsgi.input

CC'ing Peter who's in charge of Symbols on our side.
:garndt noticed that all these occurrences happened on the same machine (which got terminated meanwhile). More details can be seen here once data is updated in OrangeFactor: 
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1379155&entireHistory=true&tree=trunk 

The re-triggers are green though. In the near future, we should have monitoring tools able to tell whether a worker is consistently failing jobs and disable it. Per IRC:
"<•garndt> well the good news is that the machine is gone and retriggers are ok, so I think the issue was isoalted to that machine.  Longer term we're working on ideas of how to disable machines on the taskcluster side that pehaps would allow us to automatically throttle or disable/kill a machine that is consistently failing tasks."

Comment 4

4 months ago
7 failures in 656 pushes (0.011 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 5
* mozilla-central: 2

Platform breakdown:
* linux64: 2
* android-4-0-armv7-api15-old-id: 2
* android-4-0-armv7-api15: 2
* android-4-2-x86-old-id: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1379155&startday=2017-07-03&endday=2017-07-09&tree=all
You need to log in before you can comment on or make changes to this bug.