1057327 - Repos are being merged twice during a reconfig AND changes to tools repo are not tracked / updated in wiki, nor used to decide whether to update foopies and masters or not

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Pete Moore [:pmoore][:pete]]

In bug 1055812, the manage foopies code was moved to the end of the reconfig process, however, it is now calling function merge_to_production twice.

The problem here is that if changes occur, the version on the foopies and the version on the buildbot masters will be potentially different.

Really, we should only call this function once.

Maybe best to move it out of the if statement, and check simply [ -f "${RECONFIG_DIR}/pending_changes" ] instead of calling function.

See https://bugzilla.mozilla.org/show_bug.cgi?id=1055812#c6

Comment 1

[Pete Moore [:pmoore][:pete]]

Actually, this brings up another issue - currently if there are no changes from a merge, we don't reconfig - *however* tools repo does not have a production branch, so we should check on both masters and foopies what revision of tools we have checked out, and for the hosts that do not have the latest default changeset, we should update them.

Currently, if there is a tools change, and nothing anywhere else, the reconfig would not take place. We also would not see tools changes in the Maintenance page - so really we should show that too!

Options might be tagging when we do a reconfig, or comparing results showrevision(s) before and after reconfig to see which foopies/masters got updated, with which commits, or creating a production branch for tools repo.

Comment 2

[Chris Cooper [:coop] (he/him)]

Attachment: Apply different check for updating foopies.

(In reply to Pete Moore [:pete][:pmoore] from comment #0) 
> Maybe best to move it out of the if statement, and check simply [ -f
> "${RECONFIG_DIR}/pending_changes" ] instead of calling function.

We may have renamed this file above, so we should check for either variant.

Comment 3

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8477374 [details] [diff] [review]
Apply different check for updating foopies.

Yes, when looking at this more closely, I saw that we rename it earlier on. My original thinking was to rename after we are successful, because if we aren't successful (e.g. we exit out of reconfig due to needing to resolve a merge conflict), and then run again, we should be able to continue with the incompleted reconfig.

The problem here is that we have a single flag to say whether to reconfig, which is based on whether there were merges or not. The only repo updated on foopies is tools repo - and that doesn't get merged. So the test for updating foopies should just be if there is a tools change or not - which the pending_changes file doesn't tell us (it tells us whether there are changes to buildbot or mozharness).

Therefore, in hindsight, since we don't have an easy way to see if there are tools changes (since we don't tag tools or have a separate production branch in tools) I think the simplest is to just run it. (i.e. omit the if statement altogether). The problem with the current if statement is it is testing the wrong thing - if there were merges - and this is not related to tools changes.

Comment 4

[Pete Moore [:pmoore][:pete]]

Attachment: Update tools repo on foopies, regardless

It looks like a big diff, but it is basically just removing the if statement, and then the indenting changed. =)

Comment 5

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8477394 [details] [diff] [review]
Update tools repo on foopies, regardless

Landed: https://hg.mozilla.org/build/tools/rev/dc2af4108a34

Comment 6

[Pete Moore [:pmoore][:pete]]

So the main part left here to do is properly track changes made during reconfig, including changes to tools repo, which currently is not tracked. I would propose that we use the existing fabric action to see what versions are deployed, warn if different versions are found on different masters/foopies (i.e. they should be consistent) and then report on the full set of intermediary commits between the old version and new version per repo, potentially including links to hg diffs on hg.mozilla.org to highlight individual code line changes.

Comment 7

[Pete Moore [:pmoore][:pete]]

Doesn't make sense to fix this in light of the migration to TaskCluster.