B2G builds fail to clone files from git.mozilla.org

RESOLVED WONTFIX

Status

RESOLVED WONTFIX
3 years ago
4 months ago

People

(Reporter: aryx, Unassigned)

Tracking

Details

B2G builds are often failing because don't succeed cloning files from git.mozilla.org.

Log: https://treeherder.mozilla.org/logviewer.html#?job_id=19015475&repo=mozilla-inbound

Affected builds:
B2G ICS Emulator opt
B2G ICS Emulator debug
B2G Device Image opt
B2G Device Image debug 

[taskcluster-vcs] run end : tar -x -z -C /home/worker/workspace/B2G -f /home/worker/.tc-vcs-repo/sources/git.mozilla.org/b2g/platform_build/master.tar.gz (0) in 350 ms
[taskcluster-vcs] 0 run start : (cwd: /home/worker/workspace/gecko/testing/taskcluster/scripts/builder) tar -x -z -C /home/worker/workspace/B2G -f /home/worker/.tc-vcs-repo/sources/git.mozilla.org/b2g/fake-dalvik/master.tar.gz

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
[taskcluster-vcs:error] run end (with error) NOT RETRYING!: tar -x -z -C /home/worker/workspace/B2G -f /home/worker/.tc-vcs-repo/sources/git.mozilla.org/b2g/fake-dalvik/master.tar.gz
[taskcluster-vcs:error] Error when extracting archive

/usr/local/lib/node_modules/taskcluster-vcs/build/bin/tc-vcs.js:57
        throw err;
              ^
Error: Error running command: tar -x -z -C /home/worker/workspace/B2G -f /home/worker/.tc-vcs-repo/sources/git.mozilla.org/b2g/fake-dalvik/master.tar.gz
    at Error (<anonymous>)
    at run$ (/usr/local/lib/node_modules/taskcluster-vcs/build/run.js:128:15)
    at tryCatch (/usr/local/lib/node_modules/taskcluster-vcs/node_modules/6to5/node_modules/regenerator-6to5/runtime.js:53:40)
    at GeneratorFunctionPrototype.invoke (/usr/local/lib/node_modules/taskcluster-vcs/node_modules/6to5/node_modules/regenerator-6to5/runtime.js:209:22)
    at tryCatch (/usr/local/lib/node_modules/taskcluster-vcs/node_modules/6to5/node_modules/regenerator-6to5/runtime.js:53:40)
    at Function.step (/usr/local/lib/node_modules/taskcluster-vcs/node_modules/6to5/node_modules/regenerator-6to5/runtime.js:103:22)
    at /usr/local/lib/node_modules/taskcluster-vcs/node_modules/6to5/node_modules/core-js/shim.js:1283:41
    at /usr/local/lib/node_modules/taskcluster-vcs/node_modules/6to5/node_modules/core-js/shim.js:1293:10
    at process._tickCallback (node.js:442:13)
35 automation job failures were associated with this bug yesterday.

Repository breakdown:
* b2g-inbound: 15
* mozilla-inbound: 10
* fx-team: 10

Platform breakdown:
* b2g-device-image: 21
* b2g-emu-ics: 14

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1235196&startday=2015-12-26&endday=2015-12-26&tree=all
70 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 45
* b2g-inbound: 15
* fx-team: 10

Platform breakdown:
* b2g-device-image: 42
* b2g-emu-ics: 28

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1235196&startday=2015-12-21&endday=2015-12-27&tree=all
7 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 7

Platform breakdown:
* b2g-emu-ics: 4
* b2g-device-image: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1235196&startday=2015-12-28&endday=2016-01-03&tree=all
Hal: is this something that dev services can help investigate? Any known git issues during this timeframe?
Flags: needinfo?(hwine)
(In reply to Chris Cooper [:coop] from comment #4)
> Hal: is this something that dev services can help investigate? Any known git
> issues during this timeframe?

I'm not seeing a git.mozilla.org issue here. git.mozilla.org issues can be seen at https://status.mozilla.org/ and none show for that time period.

(In reply to Sebastian H. [:aryx][:archaeopteryx] from comment #0)
> B2G builds are often failing because don't succeed cloning files from
> git.mozilla.org.
> 
> Log:
> https://treeherder.mozilla.org/logviewer.html#?job_id=19015475&repo=mozilla-
> inbound
> 
> [taskcluster-vcs] run end : tar -x -z -C /home/worker/workspace/B2G -f
> /home/worker/.tc-vcs-repo/sources/git.mozilla.org/b2g/platform_build/master.
> tar.gz (0) in 350 ms
> [taskcluster-vcs] 0 run start : (cwd:
> /home/worker/workspace/gecko/testing/taskcluster/scripts/builder) tar -x -z
> -C /home/worker/workspace/B2G -f
> /home/worker/.tc-vcs-repo/sources/git.mozilla.org/b2g/fake-dalvik/master.tar.
> gz
> 
> gzip: stdin: not in gzip format
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
> [taskcluster-vcs:error] run end (with error) NOT RETRYING!: tar -x -z -C
> /home/worker/workspace/B2G -f
> /home/worker/.tc-vcs-repo/sources/git.mozilla.org/b2g/fake-dalvik/master.tar.
> gz

The above is not a direct operation on git.mozilla.org. AIUI, it is trying to extract a file that has been cached by task cluster. And that cache appears to be invalid.

See also bug 1204281 comment 94 for something that sounds very similar.

:garndt - is this a dupe of bug 1204281?
Flags: needinfo?(hwine) → needinfo?(garndt)

Comment 6

3 years ago
I don't think that this is a dupe.  What appears to be happening in at least bug 1204281 comment 94 is that downloading from s3 is taking a long time and timing out.  This seems to be after it already has the tarball and fails to extract.  I wonder if there was an incomplete/corrupted download or something corrupted it locally.

Looking at the last month [1] , this appears to have been a spike that since has went away but definitely should be looked into to see what measures could be taken to prevent it.

[1] https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1235196&startday=2015-12-01&endday=2016-01-05&tree=all
Flags: needinfo?(garndt)
I will resolve the bug with wontfix resolution because the problem has not been reproduced
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → WONTFIX

Updated

4 months ago
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.