Closed Bug 1512188 Opened 1 year ago Closed 1 year ago

run-task VCS options refactor

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set

Tracking

(firefox-esr60 fixed, firefox66 fixed)

RESOLVED FIXED
mozilla66
Tracking Status
firefox-esr60 --- fixed
firefox66 --- fixed

People

(Reporter: gps, Assigned: gps)

References

(Blocks 1 open bug)

Details

Attachments

(7 files, 5 obsolete files)

47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
I wrote some patches. Filing bug to track review.
We now have multiple things we may check out. "vcs" meaning "firefox"
is not obvious. Let's change the terminology to be more specific.
We have multiple source checkouts. --sparse-profile is ambiguous
as to which one it could refer to. Let's rename the argument so it
is prefixed with the repo/project we are checking out.
We currently manage VCS checkout arguments as one-offs for each
project. This isn't scalable and results in a bit of copy-pasta.

Let's make the VCS checkout arguments generic so we can have the
same control for all repositories.

This commit focuses on consolidating the existing argument parser
code. It stops short of further unification, which will be done in
subsequent commits.
This makes behavior consistent across all VCS checkouts. I'm still not
a fan of using environment variables here. But at least this gets us
1 step closer to being able to plug alternate logic in without having
to update use of environment variables outside a single function.
This is a generic normalization and doesn't need to be Firefox
specific.
This makes the call to vcs_checkout() more similar for comm checkouts,
which were previously different than others.
One step closer to making all state gathering and normalization in one
place.
Having this consistently enforced in the checkout function seems
better than looking at environment variables. Also, I think the old
logic was wrong, as it only ran if we weren't doing a checkout!
Although there is a strong possibility that I introduced this bug
via refactoring in this series.
We create a minimal wrapper function to call collect_vcs_options()
and vcs_checkout().

We could consolidate this logic into vcs_checkout(). But I don't have
strong feelings about doing that.
For historical consistency and consistency with index paths.
Attachment #9029686 - Attachment is obsolete: true
It is no longer used. We can remove it. A subsequent commit will remove
logic for handling symbolic revisions completely.
The functionality is no longer used. All CI should be pinned to a full
revision hash for determinism.
:rjl When you next update the decision task image, you'll need to figure out a way to deal with symbolic refs for m-c (which is removed here). Probably in tandem with Bug 1491371.
Flags: needinfo?(rob)
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/bc4232cfcfb8
Rename --vcs-checkout to --firefox-checkout; r=tomprince
https://hg.mozilla.org/integration/autoland/rev/4f546b3e0b18
Rename --sparse-profile to --firefox-sparse-profile; r=tomprince
https://hg.mozilla.org/integration/autoland/rev/3bcedff402fa
make VCS checkout options generic; r=tomprince
https://hg.mozilla.org/integration/autoland/rev/6bd036a5166d
Collect environment variables into VCS options; r=tomprince
https://hg.mozilla.org/integration/autoland/rev/838f49d718a7
Move base repo normalization into collect_vcs_options(); r=tomprince
https://hg.mozilla.org/integration/autoland/rev/b4f3dc9b0956
Collect hg store path in collect_vcs_options(); r=tomprince
https://hg.mozilla.org/integration/autoland/rev/d9585e5d7a3a
Move enforcement of non-symbolic revisions to vcs_checkout(); r=tomprince
https://hg.mozilla.org/integration/autoland/rev/d0eb311b3c8f
Consolidate VCS checkout from args logic; r=tomprince
https://hg.mozilla.org/integration/autoland/rev/0b3259dc10bd
Revert to "gecko" for vcs naming; r=tomprince
https://hg.mozilla.org/integration/autoland/rev/1f56f1e581e2
Remove prevent_symbolic_revisions arguments; r=tomprince
https://hg.mozilla.org/integration/autoland/rev/59813ae1b6ea
Remove support for checking out symbolic revisions; r=tomprince
Attachment #9029681 - Attachment description: Bug 1512188 - Rename --vcs-checkout to --firefox-checkout; r?tomprince → Bug 1512188 - Rename --vcs-checkout to --gecko-checkout; r?tomprince
Attachment #9029682 - Attachment description: Bug 1512188 - Rename --sparse-profile to --firefox-sparse-profile; r?tomprince → Bug 1512188 - Rename --sparse-profile to --gecko-sparse-profile; r?tomprince
Attachment #9029688 - Attachment is obsolete: true
Attachment #9029729 - Attachment is obsolete: true
Attachment #9030499 - Attachment is obsolete: true
Attachment #9030498 - Attachment is obsolete: true
Attachment #9029681 - Attachment description: Bug 1512188 - Rename --vcs-checkout to --gecko-checkout; r?tomprince → Bug 1512188 - Rename --vcs-checkout to --gecko-checkout;
Attachment #9029682 - Attachment description: Bug 1512188 - Rename --sparse-profile to --gecko-sparse-profile; r?tomprince → Bug 1512188 - Rename --sparse-profile to --gecko-sparse-profile;
Attachment #9029683 - Attachment description: Bug 1512188 - make VCS checkout options generic; r?tomprince → Bug 1512188 - make VCS checkout options generic;
Attachment #9029684 - Attachment description: Bug 1512188 - Collect environment variables into VCS options; r?tomprince → Bug 1512188 - Collect environment variables into VCS options;
Attachment #9029685 - Attachment description: Bug 1512188 - Move base repo normalization into collect_vcs_options(); r?tomprince → Bug 1512188 - Move base repo normalization into collect_vcs_options();
Attachment #9029687 - Attachment description: Bug 1512188 - Collect hg store path in collect_vcs_options(); r?tomprince → Bug 1512188 - Collect hg store path in collect_vcs_options();
Attachment #9029690 - Attachment description: Bug 1512188 - Consolidate VCS checkout from args logic; r?tomprince → Bug 1512188 - Consolidate VCS checkout from args logic;
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/autoland/rev/ac4a49eb95f5
Rename --vcs-checkout to --gecko-checkout; r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/9e95b9bf1201
Rename --sparse-profile to --gecko-sparse-profile; r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/f01d2662582f
make VCS checkout options generic; r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/f68df074eac1
Collect environment variables into VCS options; r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/ac6e2cd6eb32
Move base repo normalization into collect_vcs_options(); r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/55874a956ae1
Collect hg store path in collect_vcs_options(); r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/5e7aa7d98012
Consolidate VCS checkout from args logic; r=tomprince,dustin
Flags: needinfo?(rob)
Backed out 8 changesets (Bug 1512285, Bug 1512188) for fetch bustages.

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=5e7aa7d98012c6a21268232515f7ae2a090ee375&selectedJob=219179319

Backout link: https://hg.mozilla.org/integration/autoland/rev/d506757ef6a3eb411066b13e38f0e5bb5e18f403

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=219179319&repo=autoland&lineNumber=28

[taskcluster 2018-12-29 05:51:24.348Z] Downloading artifact "public/image.tar.zst" from task ID: FLqx-J4pQsORqABhZHwoxA.
[taskcluster 2018-12-29 05:51:25.971Z] Downloaded artifact successfully.
[taskcluster 2018-12-29 05:51:25.971Z] Downloaded 90.709 mb
[taskcluster 2018-12-29 05:51:25.971Z] Decompressing downloaded image
[taskcluster 2018-12-29 05:51:26.565Z] Loading docker image from downloaded archive.
[taskcluster 2018-12-29 05:51:37.670Z] Image 'public/image.tar.zst' from task 'FLqx-J4pQsORqABhZHwoxA' loaded.  Using image ID sha256:d64c83f3325e11403ecb81c40e3f1a78d7529599e5f7fb57efdd8bb3c31fe13d.
[taskcluster 2018-12-29 05:51:37.698Z] === Task Starting ===
[setup 2018-12-29T05:51:38.062Z] run-task started in /builds/worker
[setup 2018-12-29T05:51:38.063Z] running as worker:worker
Traceback (most recent call last):
  File "/builds/worker/bin/run-task", line 765, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/builds/worker/bin/run-task", line 745, in main
    vcs_checkout_from_args(args, 'gecko')
  File "/builds/worker/bin/run-task", line 553, in vcs_checkout_from_args
    options = collect_vcs_options(args, project)
  File "/builds/worker/bin/run-task", line 526, in collect_vcs_options
    store_path = os.environ['HG_STORE_PATH']
  File "/usr/lib/python3.5/os.py", line 725, in __getitem__
    raise KeyError(key) from None
KeyError: 'HG_STORE_PATH'
[taskcluster 2018-12-29 05:51:38.302Z] === Task Finished ===
[taskcluster 2018-12-29 05:51:38.435Z] Artifact "public" not found at "/builds/worker/artifacts"
[taskcluster 2018-12-29 05:51:39.138Z] Unsuccessful task run with exit code: 1 completed in 15.792 seconds
Flags: needinfo?(mozilla)
Flags: needinfo?(mozilla)
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/autoland/rev/862527552b97
Rename --vcs-checkout to --gecko-checkout; r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/f861ce442580
Rename --sparse-profile to --gecko-sparse-profile; r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/315bb83dabea
make VCS checkout options generic; r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/0af08435172d
Collect environment variables into VCS options; r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/504a204f8619
Move base repo normalization into collect_vcs_options(); r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/a0d4382b904b
Collect hg store path in collect_vcs_options(); r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/6d8260b9f12f
Consolidate VCS checkout from args logic; r=tomprince,dustin
Depends on: 1517177

Note that bug 1491371 saw the landing of these changes for the decision task.

You need to log in before you can comment on or make changes to this bug.