Set tooltool_cache or TOOLTOOL_CACHE for Windows builds

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: nthomas, Assigned: grenade)

Tracking

unspecified

Firefox Tracking Flags

(firefox48 fixed)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

(Reporter)

Description

2 years ago
The tooltool manifests for windows have been not-empty for some time, and the switch to VS2015 means there's quite a bit of data that can be cached locally after the first d/l. We're defining using /builds/tooltool_cache on mac and linux jobs.
(Assignee)

Comment 1

2 years ago
We can fix this in userdata golden ami runs. We already do for TaskCluster Windows worker types.
https://github.com/MozRelOps/OpenCloudConfig/blob/master/userdata/FirefoxBuildResourcesConfig.ps1#L90-L95
Assignee: nobody → rthijssen
Is the cache dir actually being used by any of the builds?
(Reporter)

Comment 3

2 years ago
I don't believe so, which is why I filed this. eg from https://treeherder.mozilla.org/logviewer.html#?job_id=19684127&repo=try:

 14:34:30     INFO - Running command: ['sh', 'c:\\builds\\moz2_slave\\try-w32-0000000000000000000000\\build\\tools\\scripts/tooltool/tooltool_wrapper.sh', 'c:\\builds\\moz2_slave\\try-w32-0000000000000000000000\\build\\src\\browser/config/tooltool-manifests/win32/releng.manifest', 'https://api.pub.build.mozilla.org/tooltool/', 'setup.sh', 'c:\\mozilla-build\\python27\\python.exe', 'C:/mozilla-build/tooltool.py', '--authentication-file', 'c:\\builds\\relengapi.tok'] in c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src
 14:34:30     INFO - Copy/paste: sh c:\builds\moz2_slave\try-w32-0000000000000000000000\build\tools\scripts/tooltool/tooltool_wrapper.sh c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\browser/config/tooltool-manifests/win32/releng.manifest https://api.pub.build.mozilla.org/tooltool/ setup.sh c:\mozilla-build\python27\python.exe C:/mozilla-build/tooltool.py --authentication-file c:\builds\relengapi.tok
 14:34:30     INFO -  c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\browser/config/tooltool-manifests/win32/releng.manifest content
 14:34:30     INFO -  =======================================================
 14:34:30     INFO -  [
 14:34:30     INFO -  {
 14:34:30     INFO -  "size": 266240,
 14:34:30     INFO -  "digest": "bb345b0e700ffab4d09436981f14b5de84da55a3f18a7f09ebc4364a4488acdeab8d46f447b12ac70f2da1444a68b8ce8b8675f0dae2ccf845e966d1df0f0869",
 14:34:30     INFO -  "algorithm": "sha512",
 14:34:30     INFO -  "filename": "mozmake.exe"
 14:34:30     INFO -  },
 14:34:30     INFO -  {
 14:34:30     INFO -  "version": "1.9.0-beta.1 (37a2869af 2016-04-12)",
 14:34:30     INFO -  "size": 96983375,
 14:34:30     INFO -  "digest": "180118e83948c24c3df2e78f02468d9011c8583c1df36d25481f74c302e9085f442133884adf431b8c05567ce1d5dfb05915cdc0bd6303d599452cad350c8de4",
 14:34:30     INFO -  "algorithm": "sha512",
 14:34:30     INFO -  "filename": "rustc.tar.bz2",
 14:34:30     INFO -  "unpack": true
 14:34:30     INFO -  },
 14:34:30     INFO -  {
 14:34:30     INFO -  "size": 167175,
 14:34:30     INFO -  "digest": "0b71a936edf5bd70cf274aaa5d7abc8f77fe8e7b5593a208f805cc9436fac646b9c4f0b43c2b10de63ff3da671497d35536077ecbc72dba7f8159a38b580f831",
 14:34:30     INFO -  "algorithm": "sha512",
 14:34:30     INFO -  "filename": "sccache.tar.bz2",
 14:34:30     INFO -  "unpack": true
 14:34:30     INFO -  },
 14:34:30     INFO -  {
 14:34:30     INFO -  "version": "Visual Studio 2015 Update 1 / SDK 10.0.10586.0",
 14:34:30     INFO -  "size": 330570496,
 14:34:30     INFO -  "digest": "0379fd087705f54aeb335449e6c623cd550b656d7110acafd1e5b315e1fc9272b7cdd1e37f99d575b16ecba4e8e4fe3af965967a3944c023b83caf68fa684888",
 14:34:30     INFO -  "algorithm": "sha512",
 14:34:30     INFO -  "filename": "vs2015u1.zip",
 14:34:30     INFO -  "unpack": true
 14:34:30     INFO -  }
 14:34:30     INFO -  ]
 14:34:30     INFO -  =======================================================
 14:34:30     INFO -  Fetching...
 14:34:31     INFO -  Executing: ['c:\\mozilla-build\\python27\\python.exe', 'C:/mozilla-build/tooltool.py', '--authentication-file', 'c:\\builds\\relengapi.tok', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--overwrite', '-m', 'c:\\builds\\moz2_slave\\try-w32-0000000000000000000000\\build\\src\\browser/config/tooltool-manifests/win32/releng.manifest', 'fetch']
 14:34:31     INFO -  INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'...
 14:34:31     INFO -  INFO - File mozmake.exe fetched from https://api.pub.build.mozilla.org/tooltool/ as c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\tmphurgtk
 14:34:31     INFO -  INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'...
 14:34:38     INFO -  INFO - File rustc.tar.bz2 fetched from https://api.pub.build.mozilla.org/tooltool/ as c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\tmp45l_aj
 14:34:38     INFO -  INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'...
 14:34:39     INFO -  INFO - File sccache.tar.bz2 fetched from https://api.pub.build.mozilla.org/tooltool/ as c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\tmp3r57qx
 14:34:39     INFO -  INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'...
 14:34:51     INFO -  INFO - File vs2015u1.zip fetched from https://api.pub.build.mozilla.org/tooltool/ as c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\tmp6f0jo4
 14:34:51     INFO -  INFO - File integrity verified, renaming tmphurgtk to mozmake.exe
 14:34:52     INFO -  INFO - File integrity verified, renaming tmp45l_aj to rustc.tar.bz2
 14:34:52     INFO -  INFO - File integrity verified, renaming tmp3r57qx to sccache.tar.bz2
 14:34:54     INFO -  INFO - File integrity verified, renaming tmp6f0jo4 to vs2015u1.zip
 14:34:54     INFO -  INFO - untarring "rustc.tar.bz2"
 14:35:05     INFO -  INFO - untarring "sccache.tar.bz2"
 14:35:05     INFO -  INFO - unzipping "vs2015u1.zip"
 14:35:23     INFO - Return code: 0

No messages there about getting content from the cache, or storing it after download+verification.
Created attachment 8743829 [details]
MozReview Request: Bug 1263575 - Add TOOLTOOL variables to windows configs r=nthomas

Review commit: https://reviewboard.mozilla.org/r/48067/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/48067/
Attachment #8743829 - Flags: review?(nthomas)
(In reply to Nick Thomas [:nthomas] from comment #3)
> I don't believe so, which is why I filed this. eg from
> https://treeherder.mozilla.org/logviewer.html#?job_id=19684127&repo=try:

I actually don't think it's used by any platforms, including windows.
(Reporter)

Comment 6

2 years ago
Comment on attachment 8743829 [details]
MozReview Request: Bug 1263575 - Add TOOLTOOL variables to windows configs r=nthomas

https://reviewboard.mozilla.org/r/48067/#review45077

::: testing/mozharness/mozharness/mozilla/building/buildbase.py:1135
(Diff revision 1)
>          ]
>          cmd.extend(c['tooltool_script'])
>          auth_file = self._get_tooltool_auth_file()
>          if auth_file:
>              cmd.extend(['--authentication-file', auth_file])
> +        cache = c['env'].get('TOOLTOOL_CACHE')

It's tempting to use a tooltool_cache variable like some other scripts do, but I think we still need TOOLTOOL_CACHE in the env for purging, so no nice things for us this week.
Attachment #8743829 - Flags: review?(nthomas) → review+
(Reporter)

Comment 7

2 years ago
(In reply to Chris AtLee [:catlee] from comment #5)
> I actually don't think it's used by any platforms, including windows.

Looks like Mac is, but Linux fails by calling like this 

18:21:17     INFO - Copy/paste: mock_mozilla -r mozilla-centos6-x86_64 -q --cwd /builds/slave/try-l64-0000000000000000000000/build/src --unpriv --shell "sh /builds/slave/try-l64-0000000000000000000000/build/tools/scripts/tooltool/tooltool_wrapper.sh /builds/slave/try-l64-0000000000000000000000/build/src/browser/config/tooltool-manifests/linux64/releng.manifest https://api.pub.build.mozilla.org/tooltool/ setup.sh /builds/tooltool.py --authentication-file /builds/relengapi.tok"

ie no environment specified.
(Reporter)

Comment 8

2 years ago
Also interesting that in your try push, this job 
 http://archive.mozilla.org/pub/firefox/try-builds/catlee@mozilla.com-6eb5f611b093745dc02508b3c9db175afd3d7b28/try-win32/try-win32-bm78-try1-build6000.txt.gz
didn't find anything precached (comment #1). A directory mismatch between /c/builds/tooltool_cache and {0}\home\worker\tooltool-cache ?

Comment 10

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/8045e15106d7
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox48: --- → fixed
Resolution: --- → FIXED
(Reporter)

Updated

2 years ago
See Also: → bug 1312650
You need to log in before you can comment on or make changes to this bug.