1512188 - run-task VCS options refactor

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Gregory Szorc [:gps]]

I wrote some patches. Filing bug to track review.

Comment 1

[Gregory Szorc [:gps]]

Attachment: Bug 1512188 - Rename --vcs-checkout to --gecko-checkout;

We now have multiple things we may check out. "vcs" meaning "firefox"
is not obvious. Let's change the terminology to be more specific.

Comment 2

[Gregory Szorc [:gps]]

Attachment: Bug 1512188 - Rename --sparse-profile to --gecko-sparse-profile;

We have multiple source checkouts. --sparse-profile is ambiguous
as to which one it could refer to. Let's rename the argument so it
is prefixed with the repo/project we are checking out.

Comment 3

[Gregory Szorc [:gps]]

Attachment: Bug 1512188 - make VCS checkout options generic;

We currently manage VCS checkout arguments as one-offs for each
project. This isn't scalable and results in a bit of copy-pasta.

Let's make the VCS checkout arguments generic so we can have the
same control for all repositories.

This commit focuses on consolidating the existing argument parser
code. It stops short of further unification, which will be done in
subsequent commits.

Comment 4

[Gregory Szorc [:gps]]

Attachment: Bug 1512188 - Collect environment variables into VCS options;

This makes behavior consistent across all VCS checkouts. I'm still not
a fan of using environment variables here. But at least this gets us
1 step closer to being able to plug alternate logic in without having
to update use of environment variables outside a single function.

Comment 5

[Gregory Szorc [:gps]]

Attachment: Bug 1512188 - Move base repo normalization into collect_vcs_options();

This is a generic normalization and doesn't need to be Firefox
specific.

Comment 6

[Gregory Szorc [:gps]]

Attachment: Bug 1512188 - Define defaults when collecting VCS options; r?tomprince

This makes the call to vcs_checkout() more similar for comm checkouts,
which were previously different than others.

Comment 7

[Gregory Szorc [:gps]]

Attachment: Bug 1512188 - Collect hg store path in collect_vcs_options();

One step closer to making all state gathering and normalization in one
place.

Comment 8

[Gregory Szorc [:gps]]

Attachment: Bug 1512188 - Move enforcement of non-symbolic revisions to vcs_checkout(); r?tomprince

Having this consistently enforced in the checkout function seems
better than looking at environment variables. Also, I think the old
logic was wrong, as it only ran if we weren't doing a checkout!
Although there is a strong possibility that I introduced this bug
via refactoring in this series.

Comment 9

[Gregory Szorc [:gps]]

Attachment: Bug 1512188 - Consolidate VCS checkout from args logic;

We create a minimal wrapper function to call collect_vcs_options()
and vcs_checkout().

We could consolidate this logic into vcs_checkout(). But I don't have
strong feelings about doing that.

Comment 10

[Gregory Szorc [:gps]]

Attachment: Bug 1512188 - Revert to "gecko" for vcs naming; r?tomprince

For historical consistency and consistency with index paths.

Comment 11

[Gregory Szorc [:gps]]

Attachment: Bug 1512188 - Remove prevent_symbolic_revisions arguments; r?tomprince

It is no longer used. We can remove it. A subsequent commit will remove
logic for handling symbolic revisions completely.

Comment 12

[Gregory Szorc [:gps]]

Attachment: Bug 1512188 - Remove support for checking out symbolic revisions; r?tomprince

The functionality is no longer used. All CI should be pinned to a full
revision hash for determinism.

Comment 13

[Tom Prince [:tomprince]]

:rjl When you next update the decision task image, you'll need to figure out a way to deal with symbolic refs for m-c (which is removed here). Probably in tandem with Bug 1491371.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bc4232cfcfb8
Rename --vcs-checkout to --firefox-checkout; r=tomprince
https://hg.mozilla.org/integration/autoland/rev/4f546b3e0b18
Rename --sparse-profile to --firefox-sparse-profile; r=tomprince
https://hg.mozilla.org/integration/autoland/rev/3bcedff402fa
make VCS checkout options generic; r=tomprince
https://hg.mozilla.org/integration/autoland/rev/6bd036a5166d
Collect environment variables into VCS options; r=tomprince
https://hg.mozilla.org/integration/autoland/rev/838f49d718a7
Move base repo normalization into collect_vcs_options(); r=tomprince
https://hg.mozilla.org/integration/autoland/rev/b4f3dc9b0956
Collect hg store path in collect_vcs_options(); r=tomprince
https://hg.mozilla.org/integration/autoland/rev/d9585e5d7a3a
Move enforcement of non-symbolic revisions to vcs_checkout(); r=tomprince
https://hg.mozilla.org/integration/autoland/rev/d0eb311b3c8f
Consolidate VCS checkout from args logic; r=tomprince
https://hg.mozilla.org/integration/autoland/rev/0b3259dc10bd
Revert to "gecko" for vcs naming; r=tomprince
https://hg.mozilla.org/integration/autoland/rev/1f56f1e581e2
Remove prevent_symbolic_revisions arguments; r=tomprince
https://hg.mozilla.org/integration/autoland/rev/59813ae1b6ea
Remove support for checking out symbolic revisions; r=tomprince

Comment 15

[Dorel Luca [:dluca]]

Backed out 12 changesets (bug 1512285, bug 1512188) for toolchain failure. CLOSED TREE

Log:
https://taskcluster-artifacts.net/cPnNKQ--Q0yTKVc9pZTfJQ/0/public/logs/live_backing.log

Push with failures:
https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&selectedJob=216431269&revision=59813ae1b6ea8d26fadc2b9e7931c26d6d965641

Backout:
https://hg.mozilla.org/integration/autoland/rev/f3fead0d1dc7e048537e288e0e21ec0246805d8d

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ac4a49eb95f5
Rename --vcs-checkout to --gecko-checkout; r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/9e95b9bf1201
Rename --sparse-profile to --gecko-sparse-profile; r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/f01d2662582f
make VCS checkout options generic; r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/f68df074eac1
Collect environment variables into VCS options; r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/ac6e2cd6eb32
Move base repo normalization into collect_vcs_options(); r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/55874a956ae1
Collect hg store path in collect_vcs_options(); r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/5e7aa7d98012
Consolidate VCS checkout from args logic; r=tomprince,dustin

Comment 17

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out 8 changesets (Bug 1512285, Bug 1512188) for fetch bustages.

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=5e7aa7d98012c6a21268232515f7ae2a090ee375&selectedJob=219179319

Backout link: https://hg.mozilla.org/integration/autoland/rev/d506757ef6a3eb411066b13e38f0e5bb5e18f403

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=219179319&repo=autoland&lineNumber=28

[taskcluster 2018-12-29 05:51:24.348Z] Downloading artifact "public/image.tar.zst" from task ID: FLqx-J4pQsORqABhZHwoxA.
[taskcluster 2018-12-29 05:51:25.971Z] Downloaded artifact successfully.
[taskcluster 2018-12-29 05:51:25.971Z] Downloaded 90.709 mb
[taskcluster 2018-12-29 05:51:25.971Z] Decompressing downloaded image
[taskcluster 2018-12-29 05:51:26.565Z] Loading docker image from downloaded archive.
[taskcluster 2018-12-29 05:51:37.670Z] Image 'public/image.tar.zst' from task 'FLqx-J4pQsORqABhZHwoxA' loaded.  Using image ID sha256:d64c83f3325e11403ecb81c40e3f1a78d7529599e5f7fb57efdd8bb3c31fe13d.
[taskcluster 2018-12-29 05:51:37.698Z] === Task Starting ===
[setup 2018-12-29T05:51:38.062Z] run-task started in /builds/worker
[setup 2018-12-29T05:51:38.063Z] running as worker:worker
Traceback (most recent call last):
  File "/builds/worker/bin/run-task", line 765, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/builds/worker/bin/run-task", line 745, in main
    vcs_checkout_from_args(args, 'gecko')
  File "/builds/worker/bin/run-task", line 553, in vcs_checkout_from_args
    options = collect_vcs_options(args, project)
  File "/builds/worker/bin/run-task", line 526, in collect_vcs_options
    store_path = os.environ['HG_STORE_PATH']
  File "/usr/lib/python3.5/os.py", line 725, in __getitem__
    raise KeyError(key) from None
KeyError: 'HG_STORE_PATH'
[taskcluster 2018-12-29 05:51:38.302Z] === Task Finished ===
[taskcluster 2018-12-29 05:51:38.435Z] Artifact "public" not found at "/builds/worker/artifacts"
[taskcluster 2018-12-29 05:51:39.138Z] Unsuccessful task run with exit code: 1 completed in 15.792 seconds

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/862527552b97
Rename --vcs-checkout to --gecko-checkout; r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/f861ce442580
Rename --sparse-profile to --gecko-sparse-profile; r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/315bb83dabea
make VCS checkout options generic; r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/0af08435172d
Collect environment variables into VCS options; r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/504a204f8619
Move base repo normalization into collect_vcs_options(); r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/a0d4382b904b
Collect hg store path in collect_vcs_options(); r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/6d8260b9f12f
Consolidate VCS checkout from args logic; r=tomprince,dustin

Comment 19

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/862527552b97
https://hg.mozilla.org/mozilla-central/rev/f861ce442580
https://hg.mozilla.org/mozilla-central/rev/315bb83dabea
https://hg.mozilla.org/mozilla-central/rev/0af08435172d
https://hg.mozilla.org/mozilla-central/rev/504a204f8619
https://hg.mozilla.org/mozilla-central/rev/a0d4382b904b
https://hg.mozilla.org/mozilla-central/rev/6d8260b9f12f

Comment 20

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/ce634269f18d
https://hg.mozilla.org/releases/mozilla-esr60/rev/654d56643a8e
https://hg.mozilla.org/releases/mozilla-esr60/rev/a2c68a828cd1
https://hg.mozilla.org/releases/mozilla-esr60/rev/2b2df02aed4c
https://hg.mozilla.org/releases/mozilla-esr60/rev/4c1c7f6eac3b
https://hg.mozilla.org/releases/mozilla-esr60/rev/8a235e8cd255
https://hg.mozilla.org/releases/mozilla-esr60/rev/cee366f3711b

Comment 21

[Dustin J. Mitchell [:dustin] (he/him)]

Note that bug 1491371 saw the landing of these changes for the decision task.