1300048 - Intermittent No extraction method found for: /home/worker/workspace/build/target.common.tests.zip

Status: RESOLVED WORKSFORME

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Seen this today:

https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=4c0092cbaf251d6253a07f103cabf8b2b1fbb9fc&filter-searchStr=fxfn&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=running&filter-resultStatus=pending&filter-resultStatus=runnable

19:42:12     INFO - Downloading packages: [u'target.common.tests.zip'] for test suite category: common
19:42:12     INFO - proxxy config: {'regions': ['.use1.', '.usw2.', '.scl3'], 'instances': ['proxxy1.srv.releng.use1.mozilla.com', 'proxxy1.srv.releng.usw2.mozilla.com', 'proxxy1.srv.releng.scl3.mozilla.com'], 'urls': [('http://ftp.mozilla.org', 'ftp.mozilla.org'), ('https://ftp.mozilla.org', 'ftp.mozilla.org'), ('https://ftp-ssl.mozilla.org', 'ftp.mozilla.org'), ('http://pypi.pvt.build.mozilla.org', 'pypi.pvt.build.mozilla.org'), ('http://pypi.pub.build.mozilla.org', 'pypi.pub.build.mozilla.org')]}
19:42:12     INFO - trying https://queue.taskcluster.net/v1/task/LuO5hf2XTMm__FaRfZKKgA/artifacts/public/build/target.common.tests.zip
19:42:12     INFO - Downloading https://queue.taskcluster.net/v1/task/LuO5hf2XTMm__FaRfZKKgA/artifacts/public/build/target.common.tests.zip to /home/worker/workspace/build/target.common.tests.zip
19:42:12     INFO - retry: Calling _download_file with args: (), kwargs: {'url': u'https://queue.taskcluster.net/v1/task/LuO5hf2XTMm__FaRfZKKgA/artifacts/public/build/target.common.tests.zip', 'file_name': u'/home/worker/workspace/build/target.common.tests.zip'}, attempt #1
19:42:12     INFO - Downloaded 1792586 bytes.
19:42:12    FATAL - No extraction method found for: /home/worker/workspace/build/target.common.tests.zip
19:42:12    FATAL - Running post_fatal callback...
19:42:12    FATAL - Exiting 2

We cannot extract the zip file because only 1792586 bytes were downloaded. I feel this is related to bug 1266624, whereby taskcluster reports the failure right away. But we fail later when trying to extract.

Comment 1

[Aki Sasaki (not active)]

It looks like download_and_extract is getting a rewrite in bug 1272083.

Comment 2

[Intermittent Failures Robot]

6 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 6

Platform breakdown:
* linux64: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1300048&startday=2016-08-29&endday=2016-09-04&tree=all

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Aki Sasaki [:aki] from comment #1)
> It looks like download_and_extract is getting a rewrite in bug 1272083.

We will see if that helped now that it got landed over the weekend.

Comment 4

[Armen [:armenzg]]

We now have a different signature.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #4)
> We now have a different signature.

Only for those cases when we download the binary or test archives. The old download method is still around and could fail similarly for test suite specific downloads.

Given that this bug has been filed first we may want to mark the others depending on it.

Btw... 

> 19:42:12     INFO - Downloaded 1792586 bytes.

This clearly indicates the download was complete. But 1.7MB for the tests.common.zip file is nothing. So the server we downloaded it from has dropped a lot of chunks.

Comment 6

[Armen [:armenzg]]

Background for garndt.
We're seeing a lot of issues spike when downloading files from TaskCluster.

garndt: is there someone that could help us look into this?

The original issue started in this bug, however, after we landed bug 1272083 the new signature is being reported in bug 1300345.
A spike is also noticeable in https://bugzilla.mozilla.org/show_bug.cgi?id=1266624#c29

Comment 7

[Greg Arndt [:garndt]]

Looking at the original bug description, it appears that mozharness tries to extract a file that was from an incomplete download.  We get this as well elsewhere within taskcluster and we compare the expected file size to what was downloaded to ensure we received all the data before trying to extract.  Is the expected and actual file sizes being compared first here?

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Those sizes aren't getting compared yet as far as I know. But in case of early aborts an exception is usually thrown. The question here is why it does not happen. This behavior reminds me on bug 1219934 where we had the same for Firefox binaries. Fixing some parts in AWS or routing stopped this problem finally. 

I really wonder if we see the same here.

And you are right, the _download method should check that the correct size has been downloaded.

Comment 9

[Armen [:armenzg]]

I will look into fixing this.

Comment 10

[Greg Arndt [:garndt]]

We also found that there were a couple of scenarios where an exception in our workers wasn't caught correctly and once that was fixed, we noticed timeout errors being thrown.  I'm not sure if this applies to the mozharness logic or not, just letting you know of something we encountered.

Comment 11

[Armen [:armenzg]]

I will investigate this on bug 1300812.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Do the responses to these requests not contain a "Content-Length" header?  Without that header, a client can't distinguish a partial from a full download (the transaction is terminated by EOF).  Comment 8 suggests that this was the issue before?  Whimboo, do you know more than was written in that bug?

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Sadly not. I never got a full description of the issue from IT. Maybe they even didn't know themselves, given that this is all EC2 magic and Amazon was not that responsive. Sorry.

Comment 14

[Armen [:armenzg]]

philor: are we getting anymore instances of this?
Or should we close since we got a new signature?

Comment 15

[Phil Ringnalda (:philor)]

A new signature in-tree, so I'd assume we would still get the old one on release branches, but I don't watch release branches and this doesn't have the intermittent-failure keyword, so I'd assume there these are just getting starred as "infra" and either retriggered or ignored.

Comment 16

[Armen [:armenzg]]

Thanks philor!

I believe we're fine.