Closed Bug 1397043 Opened 2 years ago Closed 2 years ago

Test verification downloads common tests zip more than once

Categories

(Testing :: General, defect)

defect
Not set

Tracking

(firefox56 fixed, firefox57 fixed)

RESOLVED FIXED
mozilla57
Tracking Status
firefox56 --- fixed
firefox57 --- fixed

People

(Reporter: gbrown, Assigned: gbrown)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

Oops!

https://public-artifacts.taskcluster.net/RyWMqfmbRxiZJpQX8oZSnA/0/public/logs/live_backing.log

[task 2017-09-05T10:34:50.647253Z] 10:34:50     INFO - Downloading packages: [u'target.common.tests.zip', u'target.mochitest.tests.zip'] for test suite category: mochitest
[task 2017-09-05T10:34:50.648078Z] 10:34:50     INFO - Adding 'mozinfo.json' for extraction from common.tests zip file
[task 2017-09-05T10:34:50.649126Z] 10:34:50     INFO - Downloading and extracting to /builds/worker/workspace/build/tests these dirs bin/*, certs/*, config/*, mach, marionette/*, modules/*, mozbase/*, tools/*, mochitest/*, xpcshell/*, mozinfo.json from https://queue.taskcluster.net/v1/task/Oz77JHY8R46lR-xvwFHp5Q/artifacts/public/build/target.common.tests.zip
[task 2017-09-05T10:34:50.650023Z] 10:34:50     INFO - retry: Calling fetch_url_into_memory with args: (), kwargs: {'url': u'https://queue.taskcluster.net/v1/task/Oz77JHY8R46lR-xvwFHp5Q/artifacts/public/build/target.common.tests.zip'}, attempt #1
[task 2017-09-05T10:34:50.650843Z] 10:34:50     INFO - Fetch https://queue.taskcluster.net/v1/task/Oz77JHY8R46lR-xvwFHp5Q/artifacts/public/build/target.common.tests.zip into memory
[task 2017-09-05T10:34:52.367603Z] 10:34:52     INFO - Content-Length response header: 55022230
[task 2017-09-05T10:34:52.369495Z] 10:34:52     INFO - Bytes received: 55022230
[task 2017-09-05T10:34:53.695435Z] 10:34:53     INFO - Downloading and extracting to /builds/worker/workspace/build/tests these dirs bin/*, certs/*, config/*, mach, marionette/*, modules/*, mozbase/*, tools/*, mochitest/*, xpcshell/*, mozinfo.json from https://queue.taskcluster.net/v1/task/Oz77JHY8R46lR-xvwFHp5Q/artifacts/public/build/target.mochitest.tests.zip
[task 2017-09-05T10:34:53.695790Z] 10:34:53     INFO - retry: Calling fetch_url_into_memory with args: (), kwargs: {'url': u'https://queue.taskcluster.net/v1/task/Oz77JHY8R46lR-xvwFHp5Q/artifacts/public/build/target.mochitest.tests.zip'}, attempt #1
[task 2017-09-05T10:34:53.696053Z] 10:34:53     INFO - Fetch https://queue.taskcluster.net/v1/task/Oz77JHY8R46lR-xvwFHp5Q/artifacts/public/build/target.mochitest.tests.zip into memory
[task 2017-09-05T10:34:55.452903Z] 10:34:55     INFO - Content-Length response header: 76622425
[task 2017-09-05T10:34:55.453141Z] 10:34:55     INFO - Bytes received: 76622425
[task 2017-09-05T10:35:05.005066Z] 10:35:05     INFO - Downloading packages: [u'target.common.tests.zip', u'target.xpcshell.tests.zip'] for test suite category: xpcshell
[task 2017-09-05T10:35:05.005446Z] 10:35:05     INFO - Downloading and extracting to /builds/worker/workspace/build/tests these dirs bin/*, certs/*, config/*, mach, marionette/*, modules/*, mozbase/*, tools/*, mochitest/*, xpcshell/*, mozinfo.json from https://queue.taskcluster.net/v1/task/Oz77JHY8R46lR-xvwFHp5Q/artifacts/public/build/target.common.tests.zip
[task 2017-09-05T10:35:05.005871Z] 10:35:05     INFO - retry: Calling fetch_url_into_memory with args: (), kwargs: {'url': u'https://queue.taskcluster.net/v1/task/Oz77JHY8R46lR-xvwFHp5Q/artifacts/public/build/target.common.tests.zip'}, attempt #1
[task 2017-09-05T10:35:05.006287Z] 10:35:05     INFO - Fetch https://queue.taskcluster.net/v1/task/Oz77JHY8R46lR-xvwFHp5Q/artifacts/public/build/target.common.tests.zip into memory
[task 2017-09-05T10:35:06.432410Z] 10:35:06     INFO - Content-Length response header: 55022230
[task 2017-09-05T10:35:06.432678Z] 10:35:06     INFO - Bytes received: 55022230
[task 2017-09-05T10:35:07.852182Z] 10:35:07     INFO - Downloading and extracting to /builds/worker/workspace/build/tests these dirs bin/*, certs/*, config/*, mach, marionette/*, modules/*, mozbase/*, tools/*, mochitest/*, xpcshell/*, mozinfo.json from https://queue.taskcluster.net/v1/task/Oz77JHY8R46lR-xvwFHp5Q/artifacts/public/build/target.xpcshell.tests.zip
[task 2017-09-05T10:35:07.852404Z] 10:35:07     INFO - retry: Calling fetch_url_into_memory with args: (), kwargs: {'url': u'https://queue.taskcluster.net/v1/task/Oz77JHY8R46lR-xvwFHp5Q/artifacts/public/build/target.xpcshell.tests.zip'}, attempt #1
[task 2017-09-05T10:35:07.852775Z] 10:35:07     INFO - Fetch https://queue.taskcluster.net/v1/task/Oz77JHY8R46lR-xvwFHp5Q/artifacts/public/build/target.xpcshell.tests.zip into memory
[task 2017-09-05T10:35:12.138071Z] 10:35:12     INFO - Content-Length response header: 10574837
[task 2017-09-05T10:35:12.140379Z] 10:35:12     INFO - Bytes received: 10574837
Test packages are associated with suite categories (like 'reftests'). The test-verify task has multiple suite categories, since it can run, for instance, mochitests and xpcshell tests. The common test zip is required for both mochitests and xpcshell tests, so it is included in the list of required test packages more than once.

This patch eliminates duplicates from the list before downloading, so that each package is only downloaded once.


https://treeherder.mozilla.org/#/jobs?repo=try&revision=8b4f6d49479c5a5cfee71882907bd6b7cf606375&filter-tier=1&filter-tier=2&filter-tier=3

https://public-artifacts.taskcluster.net/e4W330QASByauXFQB1Qovg/0/public/logs/live_backing.log

[task 2017-09-07T17:13:13.191314Z] 17:13:13     INFO - Downloading packages: [u'target.common.tests.zip', u'target.xpcshell.tests.zip', u'target.mochitest.tests.zip'] for test suite categories: ['mochitest', 'xpcshell']
[task 2017-09-07T17:13:13.193483Z] 17:13:13     INFO - Adding 'mozinfo.json' for extraction from common.tests zip file
[task 2017-09-07T17:13:13.199347Z] 17:13:13     INFO - Downloading and extracting to /builds/worker/workspace/build/tests these dirs bin/*, certs/*, config/*, mach, marionette/*, modules/*, mozbase/*, tools/*, mochitest/*, xpcshell/*, mozinfo.json from https://queue.taskcluster.net/v1/task/fNsejSg3ShS31ped28nNbw/artifacts/public/build/target.common.tests.zip
[task 2017-09-07T17:13:13.200288Z] 17:13:13     INFO - retry: Calling fetch_url_into_memory with args: (), kwargs: {'url': u'https://queue.taskcluster.net/v1/task/fNsejSg3ShS31ped28nNbw/artifacts/public/build/target.common.tests.zip'}, attempt #1
[task 2017-09-07T17:13:13.200946Z] 17:13:13     INFO - Fetch https://queue.taskcluster.net/v1/task/fNsejSg3ShS31ped28nNbw/artifacts/public/build/target.common.tests.zip into memory
[task 2017-09-07T17:13:14.124479Z] compiz (core) - Warn: Attempted to restack relative to 0x1600006 which is not a child of the root window or a window compiz owns
[task 2017-09-07T17:13:26.169086Z] 17:13:26     INFO - Content-Length response header: 51584400
[task 2017-09-07T17:13:26.170327Z] 17:13:26     INFO - Bytes received: 51584400
[task 2017-09-07T17:13:27.387171Z] 17:13:27     INFO - Downloading and extracting to /builds/worker/workspace/build/tests these dirs bin/*, certs/*, config/*, mach, marionette/*, modules/*, mozbase/*, tools/*, mochitest/*, xpcshell/*, mozinfo.json from https://queue.taskcluster.net/v1/task/fNsejSg3ShS31ped28nNbw/artifacts/public/build/target.xpcshell.tests.zip
[task 2017-09-07T17:13:27.388580Z] 17:13:27     INFO - retry: Calling fetch_url_into_memory with args: (), kwargs: {'url': u'https://queue.taskcluster.net/v1/task/fNsejSg3ShS31ped28nNbw/artifacts/public/build/target.xpcshell.tests.zip'}, attempt #1
[task 2017-09-07T17:13:27.388770Z] 17:13:27     INFO - Fetch https://queue.taskcluster.net/v1/task/fNsejSg3ShS31ped28nNbw/artifacts/public/build/target.xpcshell.tests.zip into memory
[task 2017-09-07T17:13:30.869397Z] 17:13:30     INFO - Content-Length response header: 13515469
[task 2017-09-07T17:13:30.870745Z] 17:13:30     INFO - Bytes received: 13515469
[task 2017-09-07T17:13:33.104032Z] 17:13:33     INFO - Downloading and extracting to /builds/worker/workspace/build/tests these dirs bin/*, certs/*, config/*, mach, marionette/*, modules/*, mozbase/*, tools/*, mochitest/*, xpcshell/*, mozinfo.json from https://queue.taskcluster.net/v1/task/fNsejSg3ShS31ped28nNbw/artifacts/public/build/target.mochitest.tests.zip
[task 2017-09-07T17:13:33.105303Z] 17:13:33     INFO - retry: Calling fetch_url_into_memory with args: (), kwargs: {'url': u'https://queue.taskcluster.net/v1/task/fNsejSg3ShS31ped28nNbw/artifacts/public/build/target.mochitest.tests.zip'}, attempt #1
[task 2017-09-07T17:13:33.105472Z] 17:13:33     INFO - Fetch https://queue.taskcluster.net/v1/task/fNsejSg3ShS31ped28nNbw/artifacts/public/build/target.mochitest.tests.zip into memory
[task 2017-09-07T17:13:39.207513Z] 17:13:39     INFO - Content-Length response header: 76741926
[task 2017-09-07T17:13:39.209332Z] 17:13:39     INFO - Bytes received: 76741926
Attachment #8905593 - Flags: review?(jmaher)
Comment on attachment 8905593 [details] [diff] [review]
eliminate duplicate packages before downloading

Review of attachment 8905593 [details] [diff] [review]:
-----------------------------------------------------------------

good find
Attachment #8905593 - Flags: review?(jmaher) → review+
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/eef129f6e50b
Update mozharness so that no test package is downloaded more than once; r=jmaher
https://hg.mozilla.org/mozilla-central/rev/eef129f6e50b
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
You need to log in before you can comment on or make changes to this bug.