Closed Bug 1379155 Opened 7 years ago Closed 2 years ago

Linux/Android Symbol upload failed

Categories

(Release Engineering :: General, enhancement)

Product:
Release Engineering
Component:
General
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: cbook, Unassigned)

References

(
URL
)

Details

like https://public-artifacts.taskcluster.net/IrvP6mJgTv6RFe9m9hxgxA/0/public/logs/live_backing.log 

Uploading symbol file "target.crashreporter-symbols-full.zip" to "https://crash-stats.mozilla.com/symbols/upload"
Attempt 1 of 5...

[taskcluster:error] Task timeout after 600 seconds. Force killing container.
[taskcluster 2017-07-07 11:06:29.834Z] === Task Finished ===
[taskcluster 2017-07-07 11:06:29.834Z] Unsuccessful task run with exit code: -1 completed in 603.179 seconds

not sure why 1/5 and what happened to 2/3/4/5 but we should investigate
also hit linux on beta
Summary: Android Symbol upload failed → Linux/Android Symbol upload failed
FWIW, this is the error we get on crash-stats: 

> UnreadablePostError
> timeout during read(65536) on wsgi.input

CC'ing Peter who's in charge of Symbols on our side.
:garndt noticed that all these occurrences happened on the same machine (which got terminated meanwhile). More details can be seen here once data is updated in OrangeFactor: 
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1379155&entireHistory=true&tree=trunk 

The re-triggers are green though. In the near future, we should have monitoring tools able to tell whether a worker is consistently failing jobs and disable it. Per IRC:
"<•garndt> well the good news is that the machine is gone and retriggers are ok, so I think the issue was isoalted to that machine.  Longer term we're working on ideas of how to disable machines on the taskcluster side that pehaps would allow us to automatically throttle or disable/kill a machine that is consistently failing tasks."

Cleanup of old bugs.

Status: NEW → RESOLVED
Closed: 2 years ago
QA Contact: aki
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.