Closed Bug 1189830 Opened 9 years ago Closed 9 years ago

Taskcluster builds often fail in tc-vcs with no usable output to Treeherder

Categories

(Taskcluster :: General, defect)

Product:
Taskcluster
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: RyanVM, Assigned: garndt)

References

Details

(Keywords: intermittent-failure) 

Spun off from bug 1147867 to get better visibility into the failure rate.
(In reply to Treeherder Robot from comment #97)
> log:
> https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-
> inbound&job_id=12421176
> repository: mozilla-inbound
> start_time: 2015-08-03T08:22:46
> who: rvandermeulen[at]mozilla[dot]com
> machine: unknown
> buildname: non-buildbot b2g-device-image test [TC] B2G Aries Debug
> revision: 10ed10f5e05e
> 
> IOError: [Errno 2] No such file or directory:
> '/home/worker/workspace/gecko/testing/taskcluster/scripts/phone-builder/
> oauth.txt'
> [/build_stage/email/index.html] [l10n] TypeError: ast[pos][attr].$v is
> undefined (app://email.gaiamobile.org)

I'm asking around to see what oauth.txt is used for and why it's missing, but it looks to only cause a warning, not anerror.  the error is the TypeError seen above.

tc-vcs seems to have cloned properly:
[tc-vcs] 0 run start : (cwd: /home/worker/workspace/gecko) hg update -C 10ed10f5e05e2f2a44e7668f49ac37f0ed7f42cc
560 files updated, 0 files merged, 34 files removed, 0 files unresolved
Blocks: 1147867
So far, the common issues appear to be:
Error: Error running command: ./repo sync -j100

Error: Error running command: tar -x -z -C /home/worker/workspace/B2G -f /home/worker/.tc-vcs-repo/sources/git.mozilla.org/external/caf/platform/prebuilts/gcc/linux-x86/arm/arm-eabi-4.7/master.tar.gz

Error: unknown type of repository (/home/worker/workspace/B2G)(this command must be run in root of repository)

The last one I've only seen once or twice.  The tar error happens the most frequently.  The tarball appears to have been downloaded, but possible there is a race between downloading it and untarring.
No longer blocks: 1147867
See Also: → 1188444
Depends on: 1194420
(In reply to Treeherder Robot from comment #1455)
> log:
> https://treeherder.mozilla.org/logviewer.html#?repo=b2g-
> inbound&job_id=2543068
> repository: b2g-inbound
> start_time: 2015-08-17T09:06:05
> who: klahnakoski[at]mozilla[dot]com
> machine: unknown
> buildname: non-buildbot mulet-linux64 test [TC] Mulet Linux
> revision: 319110cc53b8

Related to artifact upload issues
(In reply to Treeherder Robot from comment #1471)
> log:
> https://treeherder.mozilla.org/logviewer.html#?repo=b2g-
> inbound&job_id=2542058
> repository: b2g-inbound
> start_time: 2015-08-17T07:53:10
> who: klahnakoski[at]mozilla[dot]com
> machine: unknown
> buildname: non-buildbot b2g-emu-l test [TC] B2G L Emulator (Opt)
> revision: c98938b09ea7

Related to the releng api proxy not starting up
(In reply to Treeherder Robot from comment #1472)
> log:
> https://treeherder.mozilla.org/logviewer.html#?repo=b2g-
> inbound&job_id=2543820
> repository: b2g-inbound
> start_time: 2015-08-17T10:47:13
> who: klahnakoski[at]mozilla[dot]com
> machine: unknown
> buildname: non-buildbot mulet-linux64 test [TC] Mulet Linux
> revision: 3362fc869bdb

Same as comment 1474
(In reply to Treeherder Robot from comment #1615)
> log:
> https://treeherder.mozilla.org/logviewer.html#?repo=fx-team&job_id=4326671
> repository: fx-team
> start_time: 2015-08-21T03:11:58
> who: philringnalda[at]gmail[dot]com
> machine: unknown
> buildname: non-buildbot b2g-device-image test [TC] B2G Aries Eng
> revision: 62a945fe0176

Appears failures like these are from the aws cli when downloading the file.  Received chunk 97 of 131 and then tried untarring, which of course will fail if the file did not complete downloading.
With the fix of 1188444 and 1194420 the occurrence of tc-vcs related errors that do not report a proper error to TH might not be zero, but should at least be much less.  Specific bugs should be entered for tc-vcs as they are encountered.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Assignee: nobody → garndt
You need to log in before you can comment on or make changes to this bug.