Note: There are a few cases of duplicates in user autocompletion which are being worked on.

Provide a script (in desktop-build?) to fetch a single script from a gecko repo and run it

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: ted, Assigned: amiyaguchi)

Tracking

unspecified

Firefox Tracking Flags

(firefox43 fixed)

Details

Attachments

(1 attachment)

(Reporter)

Description

2 years ago
We have the checkout-sources.sh script in the desktop-build container, which makes it easy to get a full clone of a gecko repo and do things, but there are lots of tasks that it would be nice to run where we could store the script in the gecko tree (we have testing/taskcluster/scripts holding some of these), but they don't actually need a full gecko clone to run. The script I have in bug 1176229 is an example of this.

I'm toying with another script (to build minidump_stackwalk binaries) that's similar, and I'm basically writing a bootstrap script that looks like:

: GECKO_REPOSITORY              ${GECKO_REPOSITORY:=https://hg.mozilla.org/mozilla-central}
: GECKO_HEAD_REV                ${GECKO_HEAD_REV:=default}

wget ${GECKO_REPOSITORY}/raw-file/${GECKO_HEAD_REV}/testing/taskcluster/scripts/minidump_stackwalk.sh
/bin/bash minidump_stackwalk.sh

...so maybe something as simple as that would work. We could add another env var for the relative path to the script inside the repo, and then you could write a task that looked like:
```
"payload": {
  "command": [
    "./bin/run-gecko-script.sh"
  ]
  "env": {
    "GECKO_REPOSITORY": "https://hg.mozilla.org/mozilla-central",
    "GECKO_HEAD_REV": "abc123",
    "SCRIPT_PATH": "testing/taskcluster/scripts/minidump_stackwalk.sh",
  }
}
```
(Assignee)

Comment 1

2 years ago
I wouldn't mind picking up this bug, this would be relatively useful for me to implement for a one off script in bug 1168979. I've been curling the script I needed into my working directory, but it would be nicer to have a checkout-script(s).sh instead. 

I'm not sure about running the script directly though, especially in the case where the script requires arguments. In my case, I only need to pull down the script (say to $HOME/bin), so that it's available to use elsewhere.
What sort of "use elsewhere" is there except for running it?
(Assignee)

Comment 3

2 years ago
Well, I have to gather the arguments for my script (https://dxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/upload_symbols.py), but if the scripts in question are truly one off, then running them directly is probably not a bad idea.
(In reply to Anthony Miyaguchi [:amiyaguchi] from comment #3)
> Well, I have to gather the arguments for my script
> (https://dxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/
> upload_symbols.py), but if the scripts in question are truly one off, then
> running them directly is probably not a bad idea.

I think this would be useful to what you're doing. It would definitely be nice to have a canonical way to assure that we're pulling these in tree scripts from the same repo/commit as everything else. By simply curling mozilla-central, we lost the ability to test changes (the the one off script itself) with try.

I'm 100% in favor of you picking this up if you want.
(Reporter)

Comment 5

2 years ago
(In reply to Anthony Miyaguchi [:amiyaguchi] from comment #3)
> Well, I have to gather the arguments for my script
> (https://dxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/
> upload_symbols.py), but if the scripts in question are truly one off, then
> running them directly is probably not a bad idea.

I solved this in my script in comment 0 by passing parameters in as environment variables, like we do for the existing build scripts in taskcluster.

I definitely found this useful for testing--I'm able to just push changes to a user repo, trigger the task pointing GECKO_HEAD_REV at my current changeset, and have it run that copy of the script without having to fetch all of m-c.
(Assignee)

Updated

2 years ago
Assignee: nobody → amiyaguchi
QA Contact: catlee → amiyaguchi
Agreed, I think we should keep with the plan of passing parameters to task scripts via environment, even if that means that the python script gets wrapped with a shell script to re-fashion the arguments.
(Assignee)

Comment 7

2 years ago
As for the question for where such a script would go, would it be preferable for the script to live in an existing image (desktop-build), or to be in its own container?
I think we should limit the proliferation of images as much as possible, so yes, desktop-build.
(Reporter)

Comment 9

2 years ago
desktop-build seems useful, since we're talking about things running out of a gecko tree.
(Assignee)

Comment 10

2 years ago
Created attachment 8631266 [details] [diff] [review]
checkout-script.patch

This adds a checkout-script.sh to the bin directory of desktop-build, which will pull down the GECKO_HEAD_REV revision of a script from GECKO_HEAD_REPOSITORY. You can optionally specify a SCRIPT_DOWNLOAD_PATH, which will download the script to somewhere other than the current working directory.

To stay consistent with the checkout-sources.sh, checkout-script.sh will only download the script. You can run the downloaded script using a command like below.

```
"payload": {
  "command": [
    "./bin/checkout-script.sh && ./minidump_stackwalk.sh"
  ]
  "env": {
    "GECKO_REPOSITORY": "https://hg.mozilla.org/mozilla-central",
    "GECKO_HEAD_REV": "abc123",
    "SCRIPT_PATH": "testing/taskcluster/scripts/minidump_stackwalk.sh",
  }
}
```
Attachment #8631266 - Flags: review?(ted)
(Reporter)

Comment 11

2 years ago
Comment on attachment 8631266 [details] [diff] [review]
checkout-script.patch

Review of attachment 8631266 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, thanks!
Attachment #8631266 - Flags: review?(ted) → review+
(Assignee)

Updated

2 years ago
Attachment #8631266 - Flags: checked-in?
(Assignee)

Updated

2 years ago
Keywords: checkin-needed
Attachment #8631266 - Flags: checked-in?

Comment 12

2 years ago
https://hg.mozilla.org/integration/b2g-inbound/rev/a809226369ef
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/a809226369ef
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox43: --- → fixed
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.