Closed Bug 1235196 Opened 8 years ago Closed 8 years ago

B2G builds fail to clone files from git.mozilla.org

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: aryx, Unassigned)

Details

B2G builds are often failing because don't succeed cloning files from git.mozilla.org.

Log: https://treeherder.mozilla.org/logviewer.html#?job_id=19015475&repo=mozilla-inbound

Affected builds:
B2G ICS Emulator opt
B2G ICS Emulator debug
B2G Device Image opt
B2G Device Image debug 

[taskcluster-vcs] run end : tar -x -z -C /home/worker/workspace/B2G -f /home/worker/.tc-vcs-repo/sources/git.mozilla.org/b2g/platform_build/master.tar.gz (0) in 350 ms
[taskcluster-vcs] 0 run start : (cwd: /home/worker/workspace/gecko/testing/taskcluster/scripts/builder) tar -x -z -C /home/worker/workspace/B2G -f /home/worker/.tc-vcs-repo/sources/git.mozilla.org/b2g/fake-dalvik/master.tar.gz

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
[taskcluster-vcs:error] run end (with error) NOT RETRYING!: tar -x -z -C /home/worker/workspace/B2G -f /home/worker/.tc-vcs-repo/sources/git.mozilla.org/b2g/fake-dalvik/master.tar.gz
[taskcluster-vcs:error] Error when extracting archive

/usr/local/lib/node_modules/taskcluster-vcs/build/bin/tc-vcs.js:57
        throw err;
              ^
Error: Error running command: tar -x -z -C /home/worker/workspace/B2G -f /home/worker/.tc-vcs-repo/sources/git.mozilla.org/b2g/fake-dalvik/master.tar.gz
    at Error (<anonymous>)
    at run$ (/usr/local/lib/node_modules/taskcluster-vcs/build/run.js:128:15)
    at tryCatch (/usr/local/lib/node_modules/taskcluster-vcs/node_modules/6to5/node_modules/regenerator-6to5/runtime.js:53:40)
    at GeneratorFunctionPrototype.invoke (/usr/local/lib/node_modules/taskcluster-vcs/node_modules/6to5/node_modules/regenerator-6to5/runtime.js:209:22)
    at tryCatch (/usr/local/lib/node_modules/taskcluster-vcs/node_modules/6to5/node_modules/regenerator-6to5/runtime.js:53:40)
    at Function.step (/usr/local/lib/node_modules/taskcluster-vcs/node_modules/6to5/node_modules/regenerator-6to5/runtime.js:103:22)
    at /usr/local/lib/node_modules/taskcluster-vcs/node_modules/6to5/node_modules/core-js/shim.js:1283:41
    at /usr/local/lib/node_modules/taskcluster-vcs/node_modules/6to5/node_modules/core-js/shim.js:1293:10
    at process._tickCallback (node.js:442:13)
Hal: is this something that dev services can help investigate? Any known git issues during this timeframe?
Flags: needinfo?(hwine)
(In reply to Chris Cooper [:coop] from comment #4)
> Hal: is this something that dev services can help investigate? Any known git
> issues during this timeframe?

I'm not seeing a git.mozilla.org issue here. git.mozilla.org issues can be seen at https://status.mozilla.org/ and none show for that time period.

(In reply to Sebastian H. [:aryx][:archaeopteryx] from comment #0)
> B2G builds are often failing because don't succeed cloning files from
> git.mozilla.org.
> 
> Log:
> https://treeherder.mozilla.org/logviewer.html#?job_id=19015475&repo=mozilla-
> inbound
> 
> [taskcluster-vcs] run end : tar -x -z -C /home/worker/workspace/B2G -f
> /home/worker/.tc-vcs-repo/sources/git.mozilla.org/b2g/platform_build/master.
> tar.gz (0) in 350 ms
> [taskcluster-vcs] 0 run start : (cwd:
> /home/worker/workspace/gecko/testing/taskcluster/scripts/builder) tar -x -z
> -C /home/worker/workspace/B2G -f
> /home/worker/.tc-vcs-repo/sources/git.mozilla.org/b2g/fake-dalvik/master.tar.
> gz
> 
> gzip: stdin: not in gzip format
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
> [taskcluster-vcs:error] run end (with error) NOT RETRYING!: tar -x -z -C
> /home/worker/workspace/B2G -f
> /home/worker/.tc-vcs-repo/sources/git.mozilla.org/b2g/fake-dalvik/master.tar.
> gz

The above is not a direct operation on git.mozilla.org. AIUI, it is trying to extract a file that has been cached by task cluster. And that cache appears to be invalid.

See also bug 1204281 comment 94 for something that sounds very similar.

:garndt - is this a dupe of bug 1204281?
Flags: needinfo?(hwine) → needinfo?(garndt)
I don't think that this is a dupe.  What appears to be happening in at least bug 1204281 comment 94 is that downloading from s3 is taking a long time and timing out.  This seems to be after it already has the tarball and fails to extract.  I wonder if there was an incomplete/corrupted download or something corrupted it locally.

Looking at the last month [1] , this appears to have been a spike that since has went away but definitely should be looked into to see what measures could be taken to prevent it.

[1] https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1235196&startday=2015-12-01&endday=2016-01-05&tree=all
Flags: needinfo?(garndt)
I will resolve the bug with wontfix resolution because the problem has not been reproduced
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.