1179893 - Provide a script (in desktop-build?) to fetch a single script from a gecko repo and run it

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[(not currently active) Ted Mielczarek]

We have the checkout-sources.sh script in the desktop-build container, which makes it easy to get a full clone of a gecko repo and do things, but there are lots of tasks that it would be nice to run where we could store the script in the gecko tree (we have testing/taskcluster/scripts holding some of these), but they don't actually need a full gecko clone to run. The script I have in bug 1176229 is an example of this.

I'm toying with another script (to build minidump_stackwalk binaries) that's similar, and I'm basically writing a bootstrap script that looks like:

: GECKO_REPOSITORY              ${GECKO_REPOSITORY:=https://hg.mozilla.org/mozilla-central}
: GECKO_HEAD_REV                ${GECKO_HEAD_REV:=default}

wget ${GECKO_REPOSITORY}/raw-file/${GECKO_HEAD_REV}/testing/taskcluster/scripts/minidump_stackwalk.sh
/bin/bash minidump_stackwalk.sh

...so maybe something as simple as that would work. We could add another env var for the relative path to the script inside the repo, and then you could write a task that looked like:
```
"payload": {
  "command": [
    "./bin/run-gecko-script.sh"
  ]
  "env": {
    "GECKO_REPOSITORY": "https://hg.mozilla.org/mozilla-central",
    "GECKO_HEAD_REV": "abc123",
    "SCRIPT_PATH": "testing/taskcluster/scripts/minidump_stackwalk.sh",
  }
}
```

Comment 1

[Anthony Miyaguchi [:amiyaguchi]]

I wouldn't mind picking up this bug, this would be relatively useful for me to implement for a one off script in bug 1168979. I've been curling the script I needed into my working directory, but it would be nicer to have a checkout-script(s).sh instead. 

I'm not sure about running the script directly though, especially in the case where the script requires arguments. In my case, I only need to pull down the script (say to $HOME/bin), so that it's available to use elsewhere.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

What sort of "use elsewhere" is there except for running it?

Comment 3

[Anthony Miyaguchi [:amiyaguchi]]

Well, I have to gather the arguments for my script (https://dxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/upload_symbols.py), but if the scripts in question are truly one off, then running them directly is probably not a bad idea.

Comment 4

[Morgan Phillips [:mrrrgn]]

(In reply to Anthony Miyaguchi [:amiyaguchi] from comment #3)
> Well, I have to gather the arguments for my script
> (https://dxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/
> upload_symbols.py), but if the scripts in question are truly one off, then
> running them directly is probably not a bad idea.

I think this would be useful to what you're doing. It would definitely be nice to have a canonical way to assure that we're pulling these in tree scripts from the same repo/commit as everything else. By simply curling mozilla-central, we lost the ability to test changes (the the one off script itself) with try.

I'm 100% in favor of you picking this up if you want.

Comment 5

[(not currently active) Ted Mielczarek]

(In reply to Anthony Miyaguchi [:amiyaguchi] from comment #3)
> Well, I have to gather the arguments for my script
> (https://dxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/
> upload_symbols.py), but if the scripts in question are truly one off, then
> running them directly is probably not a bad idea.

I solved this in my script in comment 0 by passing parameters in as environment variables, like we do for the existing build scripts in taskcluster.

I definitely found this useful for testing--I'm able to just push changes to a user repo, trigger the task pointing GECKO_HEAD_REV at my current changeset, and have it run that copy of the script without having to fetch all of m-c.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Agreed, I think we should keep with the plan of passing parameters to task scripts via environment, even if that means that the python script gets wrapped with a shell script to re-fashion the arguments.

Comment 7

[Anthony Miyaguchi [:amiyaguchi]]

As for the question for where such a script would go, would it be preferable for the script to live in an existing image (desktop-build), or to be in its own container?

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

I think we should limit the proliferation of images as much as possible, so yes, desktop-build.

Comment 9

[(not currently active) Ted Mielczarek]

desktop-build seems useful, since we're talking about things running out of a gecko tree.

Comment 10

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: checkout-script.patch

This adds a checkout-script.sh to the bin directory of desktop-build, which will pull down the GECKO_HEAD_REV revision of a script from GECKO_HEAD_REPOSITORY. You can optionally specify a SCRIPT_DOWNLOAD_PATH, which will download the script to somewhere other than the current working directory.

To stay consistent with the checkout-sources.sh, checkout-script.sh will only download the script. You can run the downloaded script using a command like below.

```
"payload": {
  "command": [
    "./bin/checkout-script.sh && ./minidump_stackwalk.sh"
  ]
  "env": {
    "GECKO_REPOSITORY": "https://hg.mozilla.org/mozilla-central",
    "GECKO_HEAD_REV": "abc123",
    "SCRIPT_PATH": "testing/taskcluster/scripts/minidump_stackwalk.sh",
  }
}
```

Comment 11

[(not currently active) Ted Mielczarek]

Comment on attachment 8631266 [details] [diff] [review]
checkout-script.patch

Review of attachment 8631266 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, thanks!

Comment 12

[Pulsebot]

https://hg.mozilla.org/integration/b2g-inbound/rev/a809226369ef

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/a809226369ef