Closed Bug 2081250 Opened 23 days ago Closed 21 days ago

fix FIREFOX_153_5_0esr_RELEASE tag in firefox and mozilla-* repos

Categories

(Conduit :: Administration, task, P1)

Product:
Conduit
Component:
Administration
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jcristau, Assigned: shtrom)

References

(Blocks 1 open bug)

Details

Today I had to run the firefox 153.5.0esr ship phase manually to pick up the changes from bug 2046485. The builds however were coming from the FIREFOX_153_5_0esr_BUILD1 revision, from last week, and a number of uplifts landed in between. Due to bug 2081230, git commit f617beaded31d3de1cd5704e47c92811bda116b2 is what ended up being tagged as FIREFOX_153_5_0esr_RELEASE, instead of 223014c6b9299ac9b035b4c50ea481e5c6de282c.

Can we somehow delete the FIREFOX_153_5_0esr_RELEASE tag and re-tag the right revision (223014c6b9299ac9b035b4c50ea481e5c6de282c / FIREFOX_153_5_0esr_BUILD1), and mirror that over to hg?

Blocks: esr153
Assignee: nobody → omehani

I'll preface this by asking if there's any practical way we can move forwards WITHOUT doing this retagging? Could we use a different (new) tag instead?

We probably can, but this is pretty risky stuff, as we don't have a safety net. We've pushed missing tags before https://mozilla-hub.atlassian.net/wiki/spaces/EW/pages/1592754924/Git-Hg-Sync+-+Manually+pushing+a+missing+tag+to+HgMO, but never retagged.

See Also: → https://mozilla-hub.atlassian.net/wiki/spaces/EW/pages/1592754924/Git-Hg-Sync+-+Manually+pushing+a+missing+tag+to+HgMO

(In reply to Olivier Mehani [:shtrom] from comment #1)

I'll preface this by asking if there's any practical way we can move forwards WITHOUT doing this retagging? Could we use a different (new) tag instead?

I don't think there's any way we can live with this tag in its current form; there's tools and systems out there that rely on this being accurate. It would be better to have it not exist than to be incorrect IMO.

We might be able to live with it only existing in github seeing as hg is going away in the not too distant future, but even that is likely to cause problems in the near term.

(If it's any consolation, https://bugzilla.mozilla.org/show_bug.cgi?id=2081230 should be fixed very soon and this ought not to happen again...)

Plan:

  1. Delete tag FIREFOX_153_5_0esr_RELEASE git tag -d FIREFOX_153_5_0esr_RELEASE and force-push to github
  2. [git-hg-sync worker] delete the (cinnabar) tag FIREFOX_153_5_0esr_RELEASE, and push to hgmo
  3. Push the correct tag with lando-cli lando-cli push-tag --lando-repo firefox-esr153 --tag-name FIREFOX_153_5_0esr_RELEASE --tag-sha 223014c6b9299ac9b035b4c50ea481e5c6de282c

The Hg equivalent should be 3806652efdaccfd958d04803eddd3fdf131cd0cd https://hg-edge.mozilla.org/releases/mozilla-esr153/rev/FIREFOX_153_5_0esr_BUILD1

Priority: -- → P1

Will test on infra-testing first, after a sync as per bug 2016626.

See Also: → 2016626

Synced infra-testing, now pushing tags (including incorrect) with

$ git fetch --tags firefox
$ lando push-tag --lando-repo infra-testing-main

Problem: we don't have the ESR153 tags in infra-testing. Let's create test tags instead.

$ lando push-tag --lando-repo infra-testing-main --tag-name BUG_2081250_CORRECT --tag-sha $(git rev-parse main)
Working on Lando repo infra-testing-main (https://github.com/mozilla-firefox/infra-testing.git)
About to push 1 tags:
 - Tag BUG_2081250_CORRECT -> 2623d48083126c0f34255130c7107fd9292982ef
[...]
$ lando push-tag --lando-repo infra-testing-main --tag-name BUG_2081250_INCORRECT_FOR_REAL --tag-sha $(git rev-parse autoland^)
Working on Lando repo infra-testing-main (https://github.com/mozilla-firefox/infra-testing.git)
About to push 1 tags:
 - Tag BUG_2081250_INCORRECT_FOR_REAL -> d9b64d74c6037a1fbd997ff800a6dd7d0546b843

FYI, I'm primarily worried about downstreams building off the wrong revision, which becomes more likely the longer this problem exists. Can we please get this fixed ASAP?

From :zeid

We should be ready to do all the steps on DELETE_ME_3 (delete from hgmo, then GitHub, then git-hg-sync on-disk) and then re-push the same tag with this command:

lando push-tag --lando-repo infra-testing-autoland --tag-name DELETE_ME_3 --tag-sha 416e49d66aacd49d5c51619e1b6d314e3961c8d7

This will move the tag to the prior commit of what it is now. If the above does not cause any issues with git-hg-sync, we can apply those same steps to FIREFOX_153_5_0esr_RELEASE.

delete from hgmo, then GitHub, then git-hg-sync on-disk

I'd take a slightly different approach: delete in git-hg-sync on disk, and push to HgMO from there.

Git

[11:03:22] ~/work/tests/infra-testing$ git rev-parse DELETE_ME DELETE_ME_2 DELETE_ME_3
2623d48083126c0f34255130c7107fd9292982ef
2623d48083126c0f34255130c7107fd9292982ef
698f335ff82469ff3b9fb43e30c3288a27c47ad7
[11:03:29] ~/work/tests/infra-testing$ git tag -d DELETE_ME_3
Deleted tag 'DELETE_ME_3' (was 698f335ff8246)
[11:03:37] ~/work/tests/infra-testing$ git push origin :DELETE_ME_3

git-hg-sync-worker

Loosely based on tagging instructions at https://mozilla-hub.atlassian.net/wiki/spaces/EW/pages/1592754924/Git-Hg-Sync+-+Manually+pushing+a+missing+tag+to+HgMO


app@git-hg-sync-worker-0:/clones/infra-testing$ export GIT_AUTHOR_NAME="Olivier Mehani"
app@git-hg-sync-worker-0:/clones/infra-testing$ export GIT_AUTHOR_EMAIL=omehani@mozilla.com
app@git-hg-sync-worker-0:/clones/infra-testing$ export TAGS_BRANCH=tags-testing
app@git-hg-sync-worker-0:/clones/infra-testing$ export HG_REPO=ssh://hg.mozilla.org/conduit-testing/infra-testing
app@git-hg-sync-worker-0:/clones/infra-testing$ git cinnabar tag -d DELETE_ME_3 --onto refs/heads/${TAGS_BRANCH} -m 'Bug 2081250 - delete incorrect tag a=tagging CLOSED TREE DONTBUILD'  
app@git-hg-sync-worker-0:/clones/infra-testing$ /entrypoint.sh bash
[...]
app@git-hg-sync-worker-0:/clones/infra-testing$ git push hg::${HG_REPO} ${TAGS_BRANCH}:refs/heads/branches/${TAGS_BRANCH}/tip

At this point, the DELETE_ME_3 tag is no longer present in either the Git or Hg repo.

Retagging

Note that previous test tags failed to sync because they were on main, which is not synced to infra-testing on HgMO (only autoland). This means the commits were non existent, and cinnabar/Hg just silently did not tag them. This led to a fail loop, as “nothing changed” is considered an error when syncing.

[11:59:21] ~/work/tests/infra-testing$ git tag DELETE_ME_3 416e49d66aacd49d5c51619e1b6d314e3961c8d7 # This commit is `autoland^`
[12:01:36] ~/work/tests/infra-testing$ lando push-tag --lando-repo infra-testing-autoland --tag-name DELETE_ME_3 --tag-sha 416e49d66aacd49d5c51619e1b6d314e3961c8d7

Re-tagging fails because the tag already exists in the worker's work repo; this needs to be cleaned-up, too.

Lando workers

  • automationworkergit
  • landingworkergit
  • upliftworkergit

For all relevant repos-branch:

app@lando-automationworkergit-0:/files/repos$ for REPO in infra-testing-*; do git -C $REPO fetch --prune --prune-tags; done

Re-retagging

Tagging worked and was pushed successfully to the git repo.

BUT the pushlog record choked on a duplicated tag

Failed to record push log due to: duplicate key value violates unique constraint "pushlog_tag_repo_id_name_4564df3a_uniq"
DETAIL:  Key (repo_id, name)=(32, DELETE_ME_3) already exists.

So,

  • Git is in the desired state, but
  • The PushLog only has an entry for the old, undesirable state

We need the Pushlog entry to be in the correct state (i.e., have the tag point to the correct commit) to be able to trigger a sync based on the message.

We'll have to alter the pushlog...

Altering the PushLog

The admin UI purposefully forbids any pushlog modification. Because we should never have to alter it... Right? RIGHT?

$ lando shell
>>> t = Tag.objects.get(repo__name='infra-testing-autoland', name='DELETE_ME_3')
>>> t
Tag(repo=<Repo: infra-testing-autoland@autoland (git)>, name=DELETE_ME_3, commit=Commit 698f335ff82469ff3b9fb43e30c3288a27c47ad7 in infra-testing-autoland@autoland (git))
c = Commit.objects.get(repo__name='infra-testing-autoland', hash='416e49d66aacd49d5c51619e1b6d314e3961c8d7')
>>> c
Commit(repo=<Repo: infra-testing-autoland@autoland (git)>, hash=416e49d66aacd49d5c51619e1b6d314e3961c8d7)
>>> t.commit=c
>>> t
Tag(repo=<Repo: infra-testing-autoland@autoland (git)>, name=DELETE_ME_3, commit=Commit 416e49d66aacd49d5c51619e1b6d314e3961c8d7 in infra-testing-autoland@autoland (git))
>>> t.save()
>>> t.push_set.all()
<QuerySet [Push(push_id=42, repo=infra-testing-autoland@autoland (git))]>
>>> ^D
now exiting InteractiveConsole...
app@lando-web-69469dd95-htgrb:/code$ lando pulse_notify -r infra-testing-autoland -p 42 --force-renotify

That synced correctly!

And the DELETE_ME_3 tag correctly points to the Hg commit that replicate 416e from git https://hg-edge.mozilla.org/conduit-testing/infra-testing/rev/b83b5f2c40a95b699fffd9a0b53995557ae1de46

Plan

  • Tag: FIREFOX_153_5_0esr_RELEASE
  • Correct SHA: 223014c6b9299ac9b035b4c50ea481e5c6de282c (git) / 3806652efdaccfd958d04803eddd3fdf131cd0cd (hg)
  • Repo: We'll fix the tag on firefox-esr153/releases/mozilla-esr153. The Git-Hg-sync worker has a shared tags-unified branch which it updates, and pushes to all repos as new tags target them. By fixing the tag on this branch and rolling forwards, we'll ensure that all repos eventually get the updated tag without conflict.
  1. [GitHub] Delete incorrect Git Tag: FIREFOX_153_5_0esr_RELEASE in https://github.com/mozilla-firefox/firefox/tags
  2. [Lando] Clean Lando workers (landingworkergit, automationworkergit, upliftworkergit): $ cd /files/repos; for REPO in firefox-*; do echo ${REPO}; git -C ${REPO} fetch --prune --prune-tags; done
  3. [Git-Hg-Sync] Delete tag from the tags-unified branch on the syncworker-firefox and push to HgMO
$ cd /clones/firefox
$ export GIT_AUTHOR_NAME="Olivier Mehani"
$ export GIT_AUTHOR_EMAIL=omehani@mozilla.com
$ export TAG=FIREFOX_153_5_0esr_RELEASE
$ export TAGS_BRANCH=tags-unified
$ export HG_REPO=ssh://hg.mozilla.org/releases/mozilla-esr153
$ git cinnabar tag -d ${TAG} --onto refs/heads/${TAGS_BRANCH} -m "Bug 2081250 - delete incorrect tag ${FIREFOX_153_5_0esr_RELEASE} a=tagging CLOSED TREE DONTBUILD"  
$ /entrypoint.sh bash
$ git push hg::${HG_REPO} ${TAGS_BRANCH}:refs/heads/branches/${TAGS_BRANCH}/tip
  1. [Lando] Update lando pushlog: Rename the incorrect entry to avoid unicity conflicts with the updated tag.
$ lando shell
>>> repo_name='firefox-esr153'
>>> tag_name='FIREFOX_153_5_0esr_RELEASE'
>>> t = Tag.objects.get(repo__name=repo_name, name=tag_name)
>>> t.name = t.name + '_BUG_2081250'
>>> t.save()
  1. [Local] Push correct Git Tag: lando-cli push-tag --lando-repo firefox-esr153 --tag-name FIREFOX_153_5_0esr_RELEASE --tag-sha 223014c6b9299ac9b035b4c50ea481e5c6de282c.

Tag deleted from GitHub.

Uplift worker tag deletion done; took ~2 min.

app@lando-upliftworkergit-0:/files/repos$ for REPO in firefox-*; do echo $REPO; git -C $REPO tag | grep FIREFOX_153_5_0esr_RELEASE; done  
firefox-beta
firefox-esr115
firefox-esr128
firefox-esr140
firefox-esr153
firefox-release

Rest of lando workers

Landing worker done:

app@lando-landingworkergit-0:/files/repos$ for REPO in firefox-*; do echo $REPO; git -C $REPO tag | grep FIREFOX_153_5_0esr_RELEASE; done  
firefox-autoland
firefox-beta
firefox-esr115
firefox-esr128
firefox-esr140
firefox-esr153
firefox-release

Automation worker done:

$ for REPO in firefox-*; do echo $REPO; git -C $REPO tag | grep FIREFOX_153_5_0esr_RELEASE; done  
firefox-autoland
firefox-beta
firefox-esr115
firefox-esr128
firefox-esr140
firefox-esr153
firefox-main
firefox-release

Tag deleted from esr153 in HgMO https://hg-edge.mozilla.org/releases/mozilla-esr153/rev/4e1820d597eb40c7df578430dd86c2b470f2f7d8

(and I botched the commit message because ${FIREFOX_153_5_0esr_RELEASE} is not ${TAG}, the real variable containing FIREFOX_153_5_0esr_RELEASE. Oh well...

Pushlog updated

>>> repo_name='firefox-esr153'
>>> tag_name='FIREFOX_153_5_0esr_RELEASE'
>>> t = Tag.objects.get(repo__name=repo_name, name=tag_name)
>>> t
Tag(repo=<Repo: firefox-esr153@esr153 (git)>, name=FIREFOX_153_5_0esr_RELEASE, commit=Commit f617beaded31d3de1cd5704e47c92811bda116b2 in firefox-esr153@esr153 (git))
>>> t.name = t.name + '_BUG_2081250'
>>> t
Tag(repo=<Repo: firefox-esr153@esr153 (git)>, name=FIREFOX_153_5_0esr_RELEASE_BUG_2081250, commit=Commit f617beaded31d3de1cd5704e47c92811bda116b2 in firefox-esr153@esr153 (git))
>>> t.save()
Status: NEW → ASSIGNED

Tag pushed via lando with help from :abutkovits.

It landed correctly on GitHub https://github.com/mozilla-firefox/firefox/tree/FIREFOX_153_5_0esr_RELEASE

However, it did not sync to HgMO:

Tag FIREFOX_153_5_0esr_RELEASE already exists in Cinnabar, skipping for ssh://hg.mozilla.org/releases/mozilla-esr153/...

-> Need to update the cinnabar metadata, then replay the message.

Hum, no. It went through despite the warnings. This is because the warnings are shown when pushing to more than one branch: the tag is correctly created for the first branch to push, then subsequent branch processing skip creating the tag, as it's already present, but push it nonetheless.

The tag was synced to

and made its way to mozilla-unified https://hg-edge.mozilla.org/mozilla-unified/rev/f6e36402d1c56cc342f0c100d2b79e3859bb7c29

Status: ASSIGNED → RESOLVED
Closed: 21 days ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.