1474869 - mach try fuzzy doesn't work correctly with multiple checkouts

Status: RESOLVED FIXED

(Testing :: General, enhancement, P3)

Description

[James Graham [:jgraham]]

If one has two checkouts of the source tree, the task cache for mach try fuzzy isn't correctly invalidated. This is because (I assume; I didn't check the code) if tree A adds a task relative to tree B, `mach try fuzzy` in tree B will generate a cache file which appears to be valid for tree A (since the mtime will be greater the mtime of the taskcluster/ci directory in trees A and B), and therefore try builds in tree A will use the smaller taskset in tree B.

This could be fixed by caching based on the commit that last changed the tasks rather than using timestamps (although obviously supporting both git and hg) or maybe by using mtime + working tree path.

Comment 1

[Gabriel Vîjială [:gabriel-v]]

Attachment: Bug 1474869 - Move taskgraph cache files to topsrcdir/.cache.

Review commit: https://reviewboard.mozilla.org/r/260962/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/260962/

Comment 2

[Gabriel Vîjială [:gabriel-v]]

Comment on attachment 8996981 [details]
Bug 1474869 - Move taskgraph cache files to topsrcdir/.cache.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/260962/diff/1-2/

Comment 3

[Gabriel Vîjială [:gabriel-v]]

try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=f2af789df05c6546752444baeef1f3bbcba1c9d1

Comment 4

[Gabriel Vîjială [:gabriel-v]]

Attachment: Bug 1474869 - Fix mach try fuzzy when using multiple source checkouts.

Review commit: https://reviewboard.mozilla.org/r/261566/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/261566/

Comment 5

[Gabriel Vîjială [:gabriel-v]]

updated try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=9e98fc263711fc7131f863a4bca70bd7e7645dc3

Note that the commit message for that try push is wrong (says 'again' instead of 'fuzzy'), but the bug number and patch are the intended ones.

Comment 6

[Andrew Halberstadt [:ahal]]

Comment on attachment 8997854 [details]
Bug 1474869 - Fix mach try fuzzy when using multiple source checkouts.

https://reviewboard.mozilla.org/r/261566/#review269090

Lgtm.

We should probably solve the "where to store srcdir specific state" a little more generally, but that is definitely follow-up fodder.

::: tools/tryselect/tasks.py:53
(Diff revision 1)
> -    cache_dir = os.path.join(get_state_dir()[0], 'cache', 'taskgraph')
> +    root_hash = hashlib.sha256(os.path.abspath(root)).hexdigest()
> +    cache_dir = os.path.join(get_state_dir()[0], 'cache', root_hash, 'taskgraph')

Please delete the old `taskgraph` directory if it exists (no point in moving this one).

Comment 7

[Gabriel Vîjială [:gabriel-v]]

Comment on attachment 8997854 [details]
Bug 1474869 - Fix mach try fuzzy when using multiple source checkouts.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/261566/diff/1-2/

Comment 8

[Gabriel Vîjială [:gabriel-v]]

new try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a31b99ca1b90e7138ec0a850747fcadc722196aa

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4d90ecbd2cb4
Fix mach try fuzzy when using multiple source checkouts. r=ahal

Comment 10

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/4d90ecbd2cb4