Closed
Bug 1263575
Opened 8 years ago
Closed 8 years ago
Set tooltool_cache or TOOLTOOL_CACHE for Windows builds
Categories
(Release Engineering :: General, defect)
Release Engineering
General
Tracking
(firefox48 fixed)
RESOLVED
FIXED
Tracking | Status | |
---|---|---|
firefox48 | --- | fixed |
People
(Reporter: nthomas, Assigned: grenade)
References
Details
Attachments
(1 file)
The tooltool manifests for windows have been not-empty for some time, and the switch to VS2015 means there's quite a bit of data that can be cached locally after the first d/l. We're defining using /builds/tooltool_cache on mac and linux jobs.
Assignee | ||
Comment 1•8 years ago
|
||
We can fix this in userdata golden ami runs. We already do for TaskCluster Windows worker types. https://github.com/MozRelOps/OpenCloudConfig/blob/master/userdata/FirefoxBuildResourcesConfig.ps1#L90-L95
Assignee: nobody → rthijssen
Comment 2•8 years ago
|
||
Is the cache dir actually being used by any of the builds?
Reporter | ||
Comment 3•8 years ago
|
||
I don't believe so, which is why I filed this. eg from https://treeherder.mozilla.org/logviewer.html#?job_id=19684127&repo=try: 14:34:30 INFO - Running command: ['sh', 'c:\\builds\\moz2_slave\\try-w32-0000000000000000000000\\build\\tools\\scripts/tooltool/tooltool_wrapper.sh', 'c:\\builds\\moz2_slave\\try-w32-0000000000000000000000\\build\\src\\browser/config/tooltool-manifests/win32/releng.manifest', 'https://api.pub.build.mozilla.org/tooltool/', 'setup.sh', 'c:\\mozilla-build\\python27\\python.exe', 'C:/mozilla-build/tooltool.py', '--authentication-file', 'c:\\builds\\relengapi.tok'] in c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src 14:34:30 INFO - Copy/paste: sh c:\builds\moz2_slave\try-w32-0000000000000000000000\build\tools\scripts/tooltool/tooltool_wrapper.sh c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\browser/config/tooltool-manifests/win32/releng.manifest https://api.pub.build.mozilla.org/tooltool/ setup.sh c:\mozilla-build\python27\python.exe C:/mozilla-build/tooltool.py --authentication-file c:\builds\relengapi.tok 14:34:30 INFO - c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\browser/config/tooltool-manifests/win32/releng.manifest content 14:34:30 INFO - ======================================================= 14:34:30 INFO - [ 14:34:30 INFO - { 14:34:30 INFO - "size": 266240, 14:34:30 INFO - "digest": "bb345b0e700ffab4d09436981f14b5de84da55a3f18a7f09ebc4364a4488acdeab8d46f447b12ac70f2da1444a68b8ce8b8675f0dae2ccf845e966d1df0f0869", 14:34:30 INFO - "algorithm": "sha512", 14:34:30 INFO - "filename": "mozmake.exe" 14:34:30 INFO - }, 14:34:30 INFO - { 14:34:30 INFO - "version": "1.9.0-beta.1 (37a2869af 2016-04-12)", 14:34:30 INFO - "size": 96983375, 14:34:30 INFO - "digest": "180118e83948c24c3df2e78f02468d9011c8583c1df36d25481f74c302e9085f442133884adf431b8c05567ce1d5dfb05915cdc0bd6303d599452cad350c8de4", 14:34:30 INFO - "algorithm": "sha512", 14:34:30 INFO - "filename": "rustc.tar.bz2", 14:34:30 INFO - "unpack": true 14:34:30 INFO - }, 14:34:30 INFO - { 14:34:30 INFO - "size": 167175, 14:34:30 INFO - "digest": "0b71a936edf5bd70cf274aaa5d7abc8f77fe8e7b5593a208f805cc9436fac646b9c4f0b43c2b10de63ff3da671497d35536077ecbc72dba7f8159a38b580f831", 14:34:30 INFO - "algorithm": "sha512", 14:34:30 INFO - "filename": "sccache.tar.bz2", 14:34:30 INFO - "unpack": true 14:34:30 INFO - }, 14:34:30 INFO - { 14:34:30 INFO - "version": "Visual Studio 2015 Update 1 / SDK 10.0.10586.0", 14:34:30 INFO - "size": 330570496, 14:34:30 INFO - "digest": "0379fd087705f54aeb335449e6c623cd550b656d7110acafd1e5b315e1fc9272b7cdd1e37f99d575b16ecba4e8e4fe3af965967a3944c023b83caf68fa684888", 14:34:30 INFO - "algorithm": "sha512", 14:34:30 INFO - "filename": "vs2015u1.zip", 14:34:30 INFO - "unpack": true 14:34:30 INFO - } 14:34:30 INFO - ] 14:34:30 INFO - ======================================================= 14:34:30 INFO - Fetching... 14:34:31 INFO - Executing: ['c:\\mozilla-build\\python27\\python.exe', 'C:/mozilla-build/tooltool.py', '--authentication-file', 'c:\\builds\\relengapi.tok', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--overwrite', '-m', 'c:\\builds\\moz2_slave\\try-w32-0000000000000000000000\\build\\src\\browser/config/tooltool-manifests/win32/releng.manifest', 'fetch'] 14:34:31 INFO - INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'... 14:34:31 INFO - INFO - File mozmake.exe fetched from https://api.pub.build.mozilla.org/tooltool/ as c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\tmphurgtk 14:34:31 INFO - INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'... 14:34:38 INFO - INFO - File rustc.tar.bz2 fetched from https://api.pub.build.mozilla.org/tooltool/ as c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\tmp45l_aj 14:34:38 INFO - INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'... 14:34:39 INFO - INFO - File sccache.tar.bz2 fetched from https://api.pub.build.mozilla.org/tooltool/ as c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\tmp3r57qx 14:34:39 INFO - INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'... 14:34:51 INFO - INFO - File vs2015u1.zip fetched from https://api.pub.build.mozilla.org/tooltool/ as c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\tmp6f0jo4 14:34:51 INFO - INFO - File integrity verified, renaming tmphurgtk to mozmake.exe 14:34:52 INFO - INFO - File integrity verified, renaming tmp45l_aj to rustc.tar.bz2 14:34:52 INFO - INFO - File integrity verified, renaming tmp3r57qx to sccache.tar.bz2 14:34:54 INFO - INFO - File integrity verified, renaming tmp6f0jo4 to vs2015u1.zip 14:34:54 INFO - INFO - untarring "rustc.tar.bz2" 14:35:05 INFO - INFO - untarring "sccache.tar.bz2" 14:35:05 INFO - INFO - unzipping "vs2015u1.zip" 14:35:23 INFO - Return code: 0 No messages there about getting content from the cache, or storing it after download+verification.
Comment 4•8 years ago
|
||
Review commit: https://reviewboard.mozilla.org/r/48067/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/48067/
Attachment #8743829 -
Flags: review?(nthomas)
Comment 5•8 years ago
|
||
(In reply to Nick Thomas [:nthomas] from comment #3) > I don't believe so, which is why I filed this. eg from > https://treeherder.mozilla.org/logviewer.html#?job_id=19684127&repo=try: I actually don't think it's used by any platforms, including windows.
Reporter | ||
Comment 6•8 years ago
|
||
Comment on attachment 8743829 [details] MozReview Request: Bug 1263575 - Add TOOLTOOL variables to windows configs r=nthomas https://reviewboard.mozilla.org/r/48067/#review45077 ::: testing/mozharness/mozharness/mozilla/building/buildbase.py:1135 (Diff revision 1) > ] > cmd.extend(c['tooltool_script']) > auth_file = self._get_tooltool_auth_file() > if auth_file: > cmd.extend(['--authentication-file', auth_file]) > + cache = c['env'].get('TOOLTOOL_CACHE') It's tempting to use a tooltool_cache variable like some other scripts do, but I think we still need TOOLTOOL_CACHE in the env for purging, so no nice things for us this week.
Attachment #8743829 -
Flags: review?(nthomas) → review+
Reporter | ||
Comment 7•8 years ago
|
||
(In reply to Chris AtLee [:catlee] from comment #5) > I actually don't think it's used by any platforms, including windows. Looks like Mac is, but Linux fails by calling like this 18:21:17 INFO - Copy/paste: mock_mozilla -r mozilla-centos6-x86_64 -q --cwd /builds/slave/try-l64-0000000000000000000000/build/src --unpriv --shell "sh /builds/slave/try-l64-0000000000000000000000/build/tools/scripts/tooltool/tooltool_wrapper.sh /builds/slave/try-l64-0000000000000000000000/build/src/browser/config/tooltool-manifests/linux64/releng.manifest https://api.pub.build.mozilla.org/tooltool/ setup.sh /builds/tooltool.py --authentication-file /builds/relengapi.tok" ie no environment specified.
Reporter | ||
Comment 8•8 years ago
|
||
Also interesting that in your try push, this job http://archive.mozilla.org/pub/firefox/try-builds/catlee@mozilla.com-6eb5f611b093745dc02508b3c9db175afd3d7b28/try-win32/try-win32-bm78-try1-build6000.txt.gz didn't find anything precached (comment #1). A directory mismatch between /c/builds/tooltool_cache and {0}\home\worker\tooltool-cache ?
Comment 10•8 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/8045e15106d7
Updated•6 years ago
|
Component: General Automation → General
You need to log in
before you can comment on or make changes to this bug.
Description
•