Closed Bug 1963805 Opened 4 months ago Closed 3 months ago

firefox to hg.mozilla.org

Tracking

(Not tracked)

Status:

RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: shtrom)

References

(Blocks 1 open bug)

Details

(Keywords: conduit-triaged, leave-open)

Attachments

(6 files)

[mozilla-conduit/git-hg-sync] cinnabar: install latest changes from git-cinnabar upstream (Bug 1963805) (#43) 3 months ago BMO Github Automation 54 bytes, text/x-github-pull-request		Details \| Review
[mozilla-conduit/git-hg-sync] config: use different destination tag branch per target repo (bug 1963805) (#44) 3 months ago BMO Github Automation 54 bytes, text/x-github-pull-request		Details \| Review
[mozilla-conduit/git-hg-sync] repo_synchroniser: create tag branch off the first commit to be tagged (bug 1963805) (#47) 3 months ago BMO Github Automation 54 bytes, text/x-github-pull-request		Details \| Review
[mozilla-conduit/lando] pushlog_view: filter by commits- or tags-only (bug 1963805) (#327) 3 months ago BMO Github Automation 49 bytes, text/x-github-pull-request		Details \| Review
[mozilla-conduit/git-hg-sync] repo_synchroniser: don't push commits to tip to avoid non fast-forward issues (bug 1963805) (#49) 3 months ago BMO Github Automation 54 bytes, text/x-github-pull-request		Details \| Review
[mozilla-conduit/git-hg-sync] config: reenable tags in prod (bug 1963805) (#50) 3 months ago BMO Github Automation 54 bytes, text/x-github-pull-request		Details \| Review

bhearsum@mozilla.com (:bhearsum)

Reporter

Description

•

4 months ago

Examples include:
https://github.com/mozilla-firefox/firefox/releases/tag/FIREFOX-ANDROID_139_0b2_RELEASE, https://github.com/mozilla-firefox/firefox/releases/tag/FIREFOX-ANDROID_139_0b2_BUILD1, https://github.com/mozilla-firefox/firefox/releases/tag/FIREFOX_139_0b2_RELEASE, https://github.com/mozilla-firefox/firefox/releases/tag/FIREFOX_139_0b2_BUILD1 -- should be synced to https://hg.mozilla.org/releases/mozilla-beta/

https://github.com/mozilla-firefox/firefox/releases/tag/FIREFOX_138_0_1_BUILD1 - should be synced to https://hg-edge.mozilla.org/releases/mozilla-release

:glob ✱

Updated

•

4 months ago

Component: Lando → git-to-hg Sync Service

Olivier Mehani [:shtrom]

Assignee

Updated

•

4 months ago

Assignee: nobody → omehani

Status: NEW → ASSIGNED

Olivier Mehani [:shtrom]

Assignee

Updated

•

4 months ago

Severity: -- → S3

Priority: -- → P2

Olivier Mehani [:shtrom]

Assignee

Comment 1

•

4 months ago

Waiting for the fix for bug 1962924 to land, we can see the pushes with tags only using the pushlog_view command, in that they seem to be empty.

$ lando pushlog_view -r firefox-beta

[...]
Push 1000010 in firefox-beta@beta (git) (notified: True)
  Commit 044ef36b91fd97fdd3f1d3d5052b95f425d8599e in firefox-beta@beta (git)
Push 1000009 in firefox-beta@beta (git) (notified: True)
Push 1000008 in firefox-beta@beta (git) (notified: True)
Push 1000007 in firefox-beta@beta (git) (notified: True)
  Commit 5a2f6b879fa32fc0c3065bd93a13496cee024b65 in firefox-beta@beta (git)
Push 1000006 in firefox-beta@beta (git) (notified: True)
Push 1000005 in firefox-beta@beta (git) (notified: True)
Push 1000004 in firefox-beta@beta (git) (notified: True)
Push 1000003 in firefox-beta@beta (git) (notified: True)
  Commit 5fcd6333eda89bc32eca6efc9e2d21c7fe5302b0 in firefox-beta@beta (git)
  Commit df60e1f5190b092755558633eb4441e2a8d92579 in firefox-beta@beta (git)

-> A few samples are 1000004, 1000005, 1000006, 1000008, 1000009

:glob ✱

Updated

•

4 months ago

Depends on: 1962924

Olivier Mehani [:shtrom]

Assignee

Comment 2

•

4 months ago

Trying 1000007 for FIREFOX-ANDROID_139_0b2_RELEASE. This matches the currently configured pattern https://github.com/mozilla-conduit/git-hg-sync/blob/main/config-production.toml#L61-L68

>>> import re
>>> tag_pattern = "^(FIREFOX|DEVEDITION|FIREFOX-ANDROID)_(\\d+(_\\d+)+)b\\d+_(BUILD\\d+|RELEASE)$"
>>> re.match(tag_pattern, 'FIREFOX-ANDROID_139_0b2_RELEASE')
<re.Match object; span=(0, 31), match='FIREFOX-ANDROID_139_0b2_RELEASE'>

Re-sending:

$ lando pulse_notify -r firefox-beta -p 1000007 --force-renotify
Notifying for Push 1000007 in firefox-beta@beta (git) ...
{"EnvVersion": "2.0", "Fields": {"msg": "Sending {'payload': {'type': 'push', 'repo_url': 'https://github.com/mozilla-firefox/firefox.git', 'branches': {'beta': '5a2f6b879fa32fc0c3065bd93a13496cee024b65'}, 'tags': {'FIREFOX-ANDROID_139_0b2_RELEASE': '5fcd6333eda89bc32eca6efc9e2d21c7fe5302b0'}, 'time': '1746053305', 'push_id': 1000007, 'user': 'bhearsum@mozilla.com', 'push_json_url': 'FIXME', 'push_full_json_url': 'FIXME'}} ..."}, "Hostname": "lando-automationworkergit-0", "Logger": "lando", "Pid": 43072, "Severity": 6, "Timestamp": 1746172356421304320, "Type": "lando.pulse.pulse"}

Fast-forward issue

                               {"action": "log", "time": 1746172578391, "thread": "MainThread", "pid": 7, "source": "service", "component": "pulse_consumer", "message": "Received message: {'payload': {'type': 'push', 'repo_url': 'https://github.com/mozilla-firefox/firefox.git', 'branches': {'beta': '5a2f6b879fa32fc0c3065bd93a13496cee024b65'}, 'tags': {'FIREFOX-ANDROID_139_0b2_RELEASE': '5fcd6333eda89bc32eca6efc9e2d21c7fe5302b0'}, 'time': '1746053305', 'push_id': 1000007, 'user': 'bhearsum@mozilla.com', 'push_json_url': 'FIXME', 'push_full_json_url': 'FIXME'}}", "level": "INFO"}
                               {"action": "log", "time": 1746172578391, "thread": "MainThread", "pid": 7, "source": "service", "component": "pulse_consumer", "message": "parse_entity: {'type': 'push', 'repo_url': 'https://github.com/mozilla-firefox/firefox.git', 'branches': {'beta': '5a2f6b879fa32fc0c3065bd93a13496cee024b65'}, 'tags': {'FIREFOX-ANDROID_139_0b2_RELEASE': '5fcd6333eda89bc32eca6efc9e2d21c7fe5302b0'}, 'time': '1746053305', 'push_id': 1000007, 'user': 'bhearsum@mozilla.com', 'push_json_url': 'FIXME', 'push_full_json_url': 'FIXME'}", "level": "DEBUG"}
                               {"action": "log", "time": 1746172578391, "thread": "MainThread", "pid": 7, "source": "service", "component": "git_hg_sync.application", "message": "Handling push event: 1000007 for https://github.com/mozilla-firefox/firefox.git", "level": "DEBUG"}
                               {"action": "log", "time": 1746172578391, "thread": "MainThread", "pid": 7, "source": "service", "component": "sync_repo", "message": "Syncing [SyncBranchOperation(source_commit='5a2f6b879fa32fc0c3065bd93a13496cee024b65', destination_branch='default'), SyncTagOperation(source_commit='5fcd6333eda89bc32eca6efc9e2d21c7fe5302b0', tag='FIREFOX-ANDROID_139_0b2_RELEASE', tags_destination_branch='tags', tag_message_suffix='a=tagging CLOSED TREE DONTBUILD')] to ssh://hg.mozilla.org/releases/mozilla-beta/ ...", "level": "INFO"}
                               {"action": "log", "time": 1746172578393, "thread": "MainThread", "pid": 7, "source": "service", "component": "sync_repo", "message": "Cinnabar metadata already present in  /clones/firefox/.git/refs/cinnabar/metadata, not updating", "level": "DEBUG"}
                               {"action": "log", "time": 1746172578393, "thread": "MainThread", "pid": 7, "source": "service", "component": "retry", "message": "fetching source commits", "level": "DEBUG"}
2025-05-02T07:56:19.317101214Z {"action": "log", "time": 1746172579316, "thread": "MainThread", "pid": 7, "source": "service", "component": "sync_repo", "message": "AUTOLAND_REQUEST_USER set to bhearsum@mozilla.com", "level": "DEBUG"}
                               {"action": "log", "time": 1746172579317, "thread": "MainThread", "pid": 7, "source": "service", "component": "retry", "message": "adding mercurial metadata to git commits", "level": "DEBUG"}
2025-05-02T07:56:20.818602754Z {"action": "log", "time": 1746172580818, "thread": "MainThread", "pid": 7, "source": "service", "component": "retry", "message": "Failed attempt adding mercurial metadata to git commits [1/2]. Retrying...", "level": "ERROR"}
2025-05-02T07:56:22.601536522Z {"action": "log", "time": 1746172582601, "thread": "MainThread", "pid": 7, "source": "service", "component": "retry", "message": "Final attempt adding mercurial metadata to git commits [2/2] failed with error: GitCommandError: Cmd('git') failed due to: exit code(1)\n  cmdline: git -c cinnabar.data=force push --dry-run hg::ssh://hg.mozilla.org/releases/mozilla-beta/ 5a2f6b879fa32fc0c3065bd93a13496cee024b65:refs/heads/branches/default/tip\n  stderr: 'To hg::ssh://hg.mozilla.org/releases/mozilla-beta/\n ! [rejected]                    5a2f6b879fa32fc0c3065bd93a13496cee024b65 -> branches/default/tip (non-fast-forward)\nerror: failed to push some refs to 'hg::ssh://hg.mozilla.org/releases/mozilla-beta/'\nhint: Updates were rejected because a pushed branch tip is behind its remote\nhint: counterpart. Check out this branch and integrate the remote changes\nhint: (e.g. 'git pull ...') before pushing again.\nhint: See the 'Note about fast-forwards' in 'git push --help' for details.'. Aborting.", "stack": "Traceback (most recent call last):\n\n  File \"/app/git_hg_sync/retry.py\", line 20, in retry\n    callback()\n\n  File \"/app/git_hg_sync/repo_synchronizer.py\", line 103, in <lambda>\n    lambda: repo.git.execute(\n            ^^^^^^^^^^^^^^^^^\n\n  File \"/usr/local/lib/python3.12/site-packages/git/cmd.py\", line 1389, in execute\n    raise GitCommandError(redacted_command, status, stderr_value, stdout_value)\n\ngit.exc.GitCommandError: Cmd('git') failed due to: exit code(1)\n  cmdline: git -c cinnabar.data=force push --dry-run hg::ssh://hg.mozilla.org/releases/mozilla-beta/ 5a2f6b879fa32fc0c3065bd93a13496cee024b65:refs/heads/branches/default/tip\n  stderr: 'To hg::ssh://hg.mozilla.org/releases/mozilla-beta/\n ! [rejected]                    5a2f6b879fa32fc0c3065bd93a13496cee024b65 -> branches/default/tip (non-fast-forward)\nerror: failed to push some refs to 'hg::ssh://hg.mozilla.org/releases/mozilla-beta/'\nhint: Updates were rejected because a pushed branch tip is behind its remote\nhint: counterpart. Check out this branch and integrate the remote changes\nhint: (e.g. 'git pull ...') before pushing again.\nhint: See the 'Note about fast-forwards' in 'git push --help' for details.'\n", "level": "ERROR"}
                               {"action": "log", "time": 1746172582602, "thread": "MainThread", "pid": 7, "source": "service", "component": "git_hg_sync.application", "message": "An error prevented completion of the following sync operations. {\"destination_url\": \"ssh://hg.mozilla.org/releases/mozilla-beta/\", \"operations\": [{\"source_commit\": \"5a2f6b879fa32fc0c3065bd93a13496cee024b65\", \"destination_branch\": \"default\"}, {\"source_commit\": \"5fcd6333eda89bc32eca6efc9e2d21c7fe5302b0\", \"tag\": \"FIREFOX-ANDROID_139_0b2_RELEASE\", \"tags_destination_branch\": \"tags\", \"tag_message_suffix\": \"a=tagging CLOSED TREE DONTBUILD\"}]}", "stack": "Traceback (most recent call last):\n\n  File \"/app/git_hg_sync/application.py\", line 66, in _handle_push_event\n    synchronizer.sync(destination, operations, push_event.user)\n\n  File \"/app/git_hg_sync/repo_synchronizer.py\", line 101, in sync\n    retry(\n\n  File \"/app/git_hg_sync/retry.py\", line 20, in retry\n    callback()\n\n  File \"/app/git_hg_sync/repo_synchronizer.py\", line 103, in <lambda>\n    lambda: repo.git.execute(\n            ^^^^^^^^^^^^^^^^^\n\n  File \"/usr/local/lib/python3.12/site-packages/git/cmd.py\", line 1389, in execute\n    raise GitCommandError(redacted_command, status, stderr_value, stdout_value)\n\ngit.exc.GitCommandError: Cmd('git') failed due to: exit code(1)\n  cmdline: git -c cinnabar.data=force push --dry-run hg::ssh://hg.mozilla.org/releases/mozilla-beta/ 5a2f6b879fa32fc0c3065bd93a13496cee024b65:refs/heads/branches/default/tip\n  stderr: 'To hg::ssh://hg.mozilla.org/releases/mozilla-beta/\n ! [rejected]                    5a2f6b879fa32fc0c3065bd93a13496cee024b65 -> branches/default/tip (non-fast-forward)\nerror: failed to push some refs to 'hg::ssh://hg.mozilla.org/releases/mozilla-beta/'\nhint: Updates were rejected because a pushed branch tip is behind its remote\nhint: counterpart. Check out this branch and integrate the remote changes\nhint: (e.g. 'git pull ...') before pushing again.\nhint: See the 'Note about fast-forwards' in 'git push --help' for details.'\n", "level": "WARNING"}
                               {"action": "log", "time": 1746172582605, "thread": "MainThread", "pid": 7, "source": "service", "component": "pulse_consumer", "message": "Received message: {'payload': {'type': 'push', 'repo_url': 'https://github.com/mozilla-firefox/firefox.git', 'branches': {'beta': '5a2f6b879fa32fc0c3065bd93a13496cee024b65'}, 'tags': {'FIREFOX-ANDROID_139_0b2_RELEASE': '5fcd6333eda89bc32eca6efc9e2d21c7fe5302b0'}, 'time': '1746053305', 'push_id': 1000007, 'user': 'bhearsum@mozilla.com', 'push_json_url': 'FIXME', 'push_full_json_url': 'FIXME'}}", "level": "INFO"}
                               {"action": "log", "time": 1746172582605, "thread": "MainThread", "pid": 7, "source": "service", "component": "pulse_consumer", "message": "parse_entity: {'type': 'push', 'repo_url': 'https://github.com/mozilla-firefox/firefox.git', 'branches': {'beta': '5a2f6b879fa32fc0c3065bd93a13496cee024b65'}, 'tags': {'FIREFOX-ANDROID_139_0b2_RELEASE': '5fcd6333eda89bc32eca6efc9e2d21c7fe5302b0'}, 'time': '1746053305', 'push_id': 1000007, 'user': 'bhearsum@mozilla.com', 'push_json_url': 'FIXME', 'push_full_json_url': 'FIXME'}", "level": "DEBUG"}
                               {"action": "log", "time": 1746172582605, "thread": "MainThread", "pid": 7, "source": "service", "component": "git_hg_sync.application", "message": "Handling push event: 1000007 for https://github.com/mozilla-firefox/firefox.git", "level": "DEBUG"}
                               {"action": "log", "time": 1746172582605, "thread": "MainThread", "pid": 7, "source": "service", "component": "sync_repo", "message": "Syncing [SyncBranchOperation(source_commit='5a2f6b879fa32fc0c3065bd93a13496cee024b65', destination_branch='default'), SyncTagOperation(source_commit='5fcd6333eda89bc32eca6efc9e2d21c7fe5302b0', tag='FIREFOX-ANDROID_139_0b2_RELEASE', tags_destination_branch='tags', tag_message_suffix='a=tagging CLOSED TREE DONTBUILD')] to ssh://hg.mozilla.org/releases/mozilla-beta/ ...", "level": "INFO"}
                               {"action": "log", "time": 1746172582607, "thread": "MainThread", "pid": 7, "source": "service", "component": "sync_repo", "message": "Cinnabar metadata already present in  /clones/firefox/.git/refs/cinnabar/metadata, not updating", "level": "DEBUG"}
                               {"action": "log", "time": 1746172582607, "thread": "MainThread", "pid": 7, "source": "service", "component": "retry", "message": "fetching source commits", "level": "DEBUG"}
2025-05-02T07:56:23.500897915Z {"action": "log", "time": 1746172583500, "thread": "MainThread", "pid": 7, "source": "service", "component": "sync_repo", "message": "AUTOLAND_REQUEST_USER set to bhearsum@mozilla.com", "level": "DEBUG"}
                               {"action": "log", "time": 1746172583500, "thread": "MainThread", "pid": 7, "source": "service", "component": "retry", "message": "adding mercurial metadata to git commits", "level": "DEBUG"}
2025-05-02T07:56:25.070234505Z {"action": "log", "time": 1746172585070, "thread": "MainThread", "pid": 7, "source": "service", "component": "retry", "message": "Failed attempt adding mercurial metadata to git commits [1/2]. Retrying...", "level": "ERROR"}

The push contains both a commit and a tag. The commit being before the current tip, the push gets rejected as non fast-forward. An a result, the tag is not synced either.

I think we should add some logic to see if the commit is already present, and not try to push it if so.

Julien Cristau [:jcristau] (back August 18)

Comment 3

•

4 months ago

This is release-blocking, bumping severity.

Severity: S3 → S1

tracking-firefox139: --- → ?

tracking-firefox139: ? → ---

tracking-firefox140: --- → ?

tracking-firefox140: ? → ---

Priority: P2 → P1

Zeid [:zeid]

Updated

•

4 months ago

Updated

•

4 months ago

Blocks: 1963950

BMO Github Automation

Comment 4

•

3 months ago

Attached file [mozilla-conduit/git-hg-sync] cinnabar: install latest changes from git-cinnabar upstream (Bug 1963805) (#43) — Details

BMO Github Automation

Comment 5

•

3 months ago

Authored by https://github.com/cgsheeh
https://github.com/mozilla-conduit/git-hg-sync/commit/3074d26d3428c4101ff5699946f230302408cbe4
[main] cinnabar: install latest changes from git-cinnabar upstream (Bug 1963805) (#43)

Status: ASSIGNED → RESOLVED

Closed: 3 months ago

Resolution: --- → FIXED

Olivier Mehani [:shtrom]

Assignee

Updated

•

3 months ago

Status: RESOLVED → REOPENED

Keywords: leave-open

Resolution: FIXED → ---

Olivier Mehani [:shtrom]

Assignee

Comment 6

•

3 months ago

Bumping to the latest commit of git-cinnabar unclogged the sync, but we need to confirm all tags are there.

Pascal Chevrel:pascalc (PTO until Sept 2)

Comment 7

•

3 months ago

HI, I don't see updated tags on https://hg-edge.mozilla.org/releases/mozilla-beta/tags

Flags: needinfo?(omehani)

Olivier Mehani [:shtrom]

Assignee

Comment 8

•

3 months ago

Yup. I'm gonna start syncing the missing tags manually for now, as I work on the fix.

Flags: needinfo?(omehani)

BMO Github Automation

Comment 9

•

3 months ago

Attached file [mozilla-conduit/git-hg-sync] config: use different destination tag branch per target repo (bug 1963805) (#44) — Details

Julien Cristau [:jcristau] (back August 18)

Updated

•

3 months ago

Comment 10

•

3 months ago

Attached file [mozilla-conduit/git-hg-sync] repo_synchroniser: create tag branch off the first commit to be tagged (bug 1963805) (#47) — Details

:glob ✱

Updated

•

3 months ago

Blocks: hg-to-git

BMO Github Automation

Comment 11

•

3 months ago

Authored by https://github.com/shtrom
https://github.com/mozilla-conduit/git-hg-sync/commit/1a83d7c99613ea49ac873b688c17bf7d7f8a2ca0
[main] repo_synchroniser: create tag branch off the first commit to be tagged (bug 1963805) (#47)

BMO Github Automation

Comment 12

•

3 months ago

Authored by https://github.com/shtrom
https://github.com/mozilla-conduit/git-hg-sync/commit/eb31314410b85e2262b3508e67d3f627c9839b12
[main] config: use different destination tag branch per target repo (bug 1963805) (#44)

BMO Github Automation

Comment 13

•

3 months ago

Attached file [mozilla-conduit/lando] pushlog_view: filter by commits- or tags-only (bug 1963805) (#327) — Details

Olivier Mehani [:shtrom]

Assignee

Updated

•

3 months ago

Keywords: conduit-triaged

BMO Github Automation

Comment 14

•

3 months ago

Authored by https://github.com/shtrom
https://github.com/mozilla-conduit/lando/commit/8fb5b9444e4e29a0b1fb6a21b98234bd7a81e44d
[main] pushlog_view: filter by commits- or tags-only (bug 1963805) (#327)

BMO Github Automation

Comment 15

•

3 months ago

Attached file [mozilla-conduit/git-hg-sync] repo_synchroniser: don't push commits to tip to avoid non fast-forward issues (bug 1963805) (#49) — Details

Cosmin Sabou [:CosminS]

Updated

•

3 months ago

Comment 16

•

3 months ago

Attached file [mozilla-conduit/git-hg-sync] config: reenable tags in prod (bug 1963805) (#50) — Details

BMO Github Automation

Comment 17

•

3 months ago

Authored by https://github.com/shtrom
https://github.com/mozilla-conduit/git-hg-sync/commit/92da36ae3dad7f9a6040a33fb98618c737059d6b
[main] config: reenable tags in prod (bug 1963805) (#50)

Olivier Mehani [:shtrom]

Assignee

Comment 18

•

3 months ago

•

Edited

I'm closing this bug, now that we have reenabled the tag syncing in the config on a proper release.

There is outstanding work in github.com/mozilla-conduit/git-hg-sync/pull/49, which was necessary for the one-off resync, but not strictly for moving forwards with syncing in a stable situation. I'll split this to it's own ticket.

edit: updated PR URL

Status: REOPENED → RESOLVED

Closed: 3 months ago → 3 months ago

Resolution: --- → FIXED

Olivier Mehani [:shtrom]

Assignee

Updated

•

3 months ago