end_to_end_reconfig.sh should use tip revision of production-masters.json and devices.json rather than (potentially dated) locally checked out version

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
3 years ago
5 months ago

People

(Reporter: pmoore, Assigned: pmoore)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

3 years ago
The problem comes when you run end_to_end_reconfig.sh on your local machine, but you forgot to first update the tools repo where this is checked out. If there are changes to production-masters.json or devices.json, you are not going to get them, and this can be devastating (i.e. not all masters or foopies are updated, or process can crash).

Instead, we should use the tip version of both of these files, to make sure we get the absolute latest versions, regardless of what is locally cached in the local checkout. This also forces us to push local changes to these files before running the reconfig, which is also a good safety net, as the changes get tracked.

Worse case, this can be overridden if for some reason we don't want to push, by hand editing the end_to_end_reconfig.sh script, but hopefully there shouldn't ever be a need to do that.
(Assignee)

Comment 1

3 years ago
Created attachment 8453692 [details] [diff] [review]
bug1036874_tools_v1.patch

Hey Jordan,

I spotted here: https://wiki.mozilla.org/ReleaseEngineering/Managing_Buildbot_with_Fabric#Usage that a url can also be used with the -f option, so I figured this might be the way to go. I considered checking out full tools repo, so we'd also get latest version of end_to_end_reconfig.sh and update_maintenance_wiki.sh etc - but then I thought that might not be ideal if we are testing changes to them, and then if scripts update themselves as they are running, ... well ... it gets messy and complicated. So I think this probably a more suitable approach, just getting the latest config files, as they are the most important, and we don't want to assume that the user hasn't customised the local version of the scripts to work around an issue etc.

Many thanks in advance!
Pete
Attachment #8453692 - Flags: review?(jlund)

Comment 2

3 years ago
Comment on attachment 8453692 [details] [diff] [review]
bug1036874_tools_v1.patch

Review of attachment 8453692 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm :)

logic seems sound for why you went with this approach. plus, we can get rid of all that ugly sed regex patterns ;)

::: buildfarm/maintenance/end_to_end_reconfig.sh
@@ +352,5 @@
> +        echo "  * Preparing reconfig only; not running: '$(pwd)/manage_masters.py' -f '${production_masters_url}' -j16 -R scheduler -R build -R try -R tests show_revisions update"
> +        echo "  * Preparing reconfig only; not running: '$(pwd)/manage_foopies.py' -f '${devices_json_url}' -j16 -H all show_revision update"
> +        echo "  * Preparing reconfig only; not running: '$(pwd)/manage_masters.py' -f '${production_masters_url}' -j32 -R scheduler -R build -R try -R tests checkconfig reconfig"
> +        echo "  * Preparing reconfig only; not running: '$(pwd)/manage_masters.py' -f '${production_masters_url}' -j16 -R scheduler -R build -R try -R tests show_revisions"
> +        echo "  * Preparing reconfig only; not running: '$(pwd)/manage_foopies.py' -f '${devices_json_url}' -j16 -H all show_revision"

OOC any reason we changed to 16 here?
Attachment #8453692 - Flags: review?(jlund) → review+
(Assignee)

Comment 3

3 years ago
(In reply to Jordan Lund (:jlund) from comment #2)
> Comment on attachment 8453692 [details] [diff] [review]
> bug1036874_tools_v1.patch
> 
> Review of attachment 8453692 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> lgtm :)
> 
> logic seems sound for why you went with this approach. plus, we can get rid
> of all that ugly sed regex patterns ;)
> 
> ::: buildfarm/maintenance/end_to_end_reconfig.sh
> @@ +352,5 @@
> > +        echo "  * Preparing reconfig only; not running: '$(pwd)/manage_masters.py' -f '${production_masters_url}' -j16 -R scheduler -R build -R try -R tests show_revisions update"
> > +        echo "  * Preparing reconfig only; not running: '$(pwd)/manage_foopies.py' -f '${devices_json_url}' -j16 -H all show_revision update"
> > +        echo "  * Preparing reconfig only; not running: '$(pwd)/manage_masters.py' -f '${production_masters_url}' -j32 -R scheduler -R build -R try -R tests checkconfig reconfig"
> > +        echo "  * Preparing reconfig only; not running: '$(pwd)/manage_masters.py' -f '${production_masters_url}' -j16 -R scheduler -R build -R try -R tests show_revisions"
> > +        echo "  * Preparing reconfig only; not running: '$(pwd)/manage_foopies.py' -f '${devices_json_url}' -j16 -H all show_revision"
> 
> OOC any reason we changed to 16 here?

Thanks Jordan!

I changed from 15 to 16 just to be consistent with the other lines (elsewhere it was all powers of 2).

I, personally, will miss the sed regexes... (http://uuner.doslash.org/forfun/sedtris.sed has always been my inspiration)
(Assignee)

Comment 4

3 years ago
Comment on attachment 8453692 [details] [diff] [review]
bug1036874_tools_v1.patch

https://hg.mozilla.org/build/tools/rev/678c93d3ea62
Attachment #8453692 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Component: Tools → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.