mach try lando fails when normal mach try succeeds
Categories
(Developer Infrastructure :: Try, defect)
Tracking
(firefox123 affected)
| Tracking | Status | |
|---|---|---|
| firefox123 | --- | affected |
People
(Reporter: Gijs, Assigned: sheehan)
References
(Blocks 2 open bugs)
Details
Attachments
(2 files)
This was my apparently successful push via lando:
% ./mach try fuzzy --preset desktop-frontend --no-artifact --push-to-lando
estimates: Runs 160 tasks (128 selected, 32 dependencies)
estimates: Total task duration 2 days, 5:24:20
estimates: In the top 22% of durations
estimates: Should take about 1:38:42 (Finished around 2023-10-03 13:06)
No existing Auth0 token found.
<snip auth>
Waiting.......
Login successful.
Auth0 token validated.
Using 53af11a26cb69e0616d9f481f988276e0a6da93e as the hg base commit.
Submitting stack of 3 nodes.
Patches gathered for submission.
Submitting patches to Lando.
Lando try submission success, took 14.81 seconds. Landing job id: 67873.
Then I got email:
Your request to land Bug 1855992 - remove about: specialcase from principal validation in BrowserParent, r?nika! failed.
See https://lando.services.mozilla.com/Bug 1855992 - remove about: specialcase from principal validation in BrowserParent, r?nika!/ for details.
Reason:
Problem while applying patch in revision None:
hg error in cmd: hg import --no-commit -s 95 /tmp/tmpj7z8s1te: applying /tmp/tmpj7z8s1te
patching file toolkit/mozapps/extensions/test/browser/browser.toml
Hunk #1 FAILED at 9
1 out of 1 hunks FAILED -- saving rejects to file toolkit/mozapps/extensions/test/browser/browser.toml.rej
abort: patch failed to apply
I then retried without lando and got:
% ./mach try fuzzy --preset desktop-frontend --no-artifact
estimates: Runs 160 tasks (128 selected, 32 dependencies)
estimates: Total task duration 2 days, 5:24:20
estimates: In the top 22% of durations
estimates: Should take about 1:38:42 (Finished around 2023-10-03 13:34)
Creating temporary commit for remote...
A try_task_config.json
pushing to ssh://hg.mozilla.org/try
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: recorded push in pushlog
remote: added 3 changesets with 7 changes to 26 files (+1 heads)
remote:
remote: View your changes here:
remote: https://hg.mozilla.org/try/rev/b4b921049973dafdbb80e5accaecbace770cbf60
remote: https://hg.mozilla.org/try/rev/0ce19111852cedaf83c79657d641ee4a6e22caf5
remote: https://hg.mozilla.org/try/rev/92927dc502ed3806da76711c1e4a1d59e57fba71
remote:
remote: Follow the progress of your build on Treeherder:
remote: https://treeherder.mozilla.org/jobs?repo=try&revision=92927dc502ed3806da76711c1e4a1d59e57fba71
remote: recorded changegroup in replication log in 0.019s
push complete
temporary commit removed, repository restored
There are a few things that are confusing here.
- why didn't it "just work"? I submitted 2 commits on top of m-c from this morning. If lando is trying to reapply them on top of autoland then that won't work if any of the files have changed, but that's not how I'd expect this to work... Especially when doing e.g. performance comparison trypushes, it's quite important that the base revision is the same between multiple trypushes.
- the message says "Problem while applying patch in revision None:" which sounds ominous/wrong.
- it would be nice if the email subject and message clarified this was a trypush vs. a "real" land-on-autoland lando.
|
Reporter | |
Comment 1•2 years ago
|
||
(In reply to :Gijs (he/him) from comment #0)
- it would be nice if the email subject and message clarified this was a trypush vs. a "real" land-on-autoland lando.
Looks like this part may be related to bug 1855851, though not quite the same.
|
||
Comment 3•2 years ago
|
||
I'm also hitting this issue. That's surprising as it fails to apply the patch on files, which, I think, aren't being modified anytime recently.
Info about the push to lando:
Using ca60a5861d4a8bd5de1c8e5a9f76c9e35767c38c as the hg base commit.
Lando try submission success, took 2.31 seconds. Landing job id: 69158.
Failure message by email:
Your request to land Bug 1848179 - [devtools] Display the currently processed DOM event for top frames. failed.
See https://lando.services.mozilla.com/Bug 1848179 - [devtools] Display the currently processed DOM event for top frames./ for details.
Reason:
Problem while applying patch in revision None:
hg error in cmd: hg import --no-commit -s 95 /tmp/tmp5mcfvpiw: applying /tmp/tmp5mcfvpiw
patching file devtools/client/debugger/test/mochitest/browser_dbg-javascript-tracer.js
Hunk #2 FAILED at 120
1 out of 2 hunks FAILED -- saving rejects to file devtools/client/debugger/test/mochitest/browser_dbg-javascript-tracer.js.rej
patching file devtools/server/actors/tracer.js
Hunk #4 succeeded at 157 with fuzz 2 (offset 0 lines).
patching file devtools/server/tracer/tracer.jsm
Hunk #7 FAILED at 293
1 out of 7 hunks FAILED -- saving rejects to file devtools/server/tracer/tracer.jsm.rej
abort: patch failed to apply
Functional push to try:
https://treeherder.mozilla.org/jobs?repo=try&revision=388cedb1f05198de7e05f05d8a06fd3c32627fac
Changeset on phabricator:
https://phabricator.services.mozilla.com/D185934
|
||
Comment 4•2 years ago
|
||
I just hit this as well. What's the model here? How is Lando choosing the tip to apply the patches to? How is this intended to work?
|
Assignee | |
Comment 5•2 years ago
|
||
(In reply to Nick Alexander :nalexander [he/him] from comment #4)
I just hit this as well. What's the model here? How is Lando choosing the tip to apply the patches to? How is this intended to work?
It applies patches to the base Mercurial commit SHA displayed in the terminal. On Git, that commit is found by looking for the first commit not present on a remote branch with git rev-list HEAD --topo-order --boundary --not --remotes that is then passed to git cinnabar git2hg. On Mercurial we look for the first public ancestor in the commit stack.
Patches are gathered with hg export in Mercurial and git format-patch in Git. We parse the diff from the patch on the Lando side and apply it directly in the same fashion as we do for Phabricator patches.
I'm going to spend some time in the next few days working on fixing this bug.
|
|
||
Comment 6•2 years ago
|
||
(In reply to Connor Sheehan [:sheehan] from comment #5)
(In reply to Nick Alexander :nalexander [he/him] from comment #4)
I just hit this as well. What's the model here? How is Lando choosing the tip to apply the patches to? How is this intended to work?
It applies patches to the base Mercurial commit SHA displayed in the terminal. On Git, that commit is found by looking for the first commit not present on a remote branch with
git rev-list HEAD --topo-order --boundary --not --remotesthat is then passed togit cinnabar git2hg. On Mercurial we look for the first public ancestor in the commit stack.Patches are gathered with
hg exportin Mercurial andgit format-patchin Git. We parse the diff from the patch on the Lando side and apply it directly in the same fashion as we do for Phabricator patches.
The push that failed for me said:
Your request to land Bug 1850149 - Part 2: WIP on testing telemetry. failed.
See https://lando.services.mozilla.com/Bug 1850149 - Part 2: WIP on testing telemetry./ for details.
Reason:
Problem while applying patch in revision None:
hg error in cmd: hg import --no-commit -s 95 /tmp/tmppae3gcdl: applying /tmp/tmppae3gcdl
patching file toolkit/mozapps/defaultagent/metrics.yaml
Hunk #1 FAILED at 9
1 out of 1 hunks FAILED -- saving rejects to file toolkit/mozapps/defaultagent/metrics.yaml.rej
abort: patch failed to apply
I'm going to spend some time in the next few days working on fixing this bug.
Great! As an aside, could you include an ID and a well-formed URL in the message? The existing one is not correct, presumably it's supposed to be DXXXX and not part of the commit message.
|
|
Assignee | |
Comment 7•2 years ago
|
||
(In reply to Nick Alexander :nalexander [he/him] from comment #6)
Great! As an aside, could you include an ID and a well-formed URL in the message? The existing one is not correct, presumably it's supposed to be
DXXXXand not part of the commit message.
Certainly, I'll tackle that work in bug 1855851. :)
|
||
Comment 8•2 years ago
|
||
On Git, that commit is found by looking for the first commit not present on a remote branch with git rev-list HEAD --topo-order --boundary --not --remotes that is then passed to git cinnabar git2hg.
This doesn't work when you have a remote for a project branch, and your local branch is derived from it.
On Mercurial we look for the first public ancestor in the commit stack.
This probably has the same caveat.
And while that doesn't explain comment 0, that /could/ explain some of the subsequent reports.
|
|
Assignee | |
Comment 9•2 years ago
|
||
(In reply to Mike Hommey [:glandium] from comment #8)
This doesn't work when you have a remote for a project branch, and your local branch is derived from it.
Someone has reported this case already, but the error they saw was abort: unknown revision '9e801e3051d94fb4f33a0bc05fe40254d39f7e5f', since the revision that was passed as the base revision to apply the patches to isn't found in mozilla-unified during repo setup in Lando. The failures in this bug are happening during patch application, so the base commit should be confirmed to exist in unified by the time we see this bug.
You're right that having another remote branch that shares commits with your current branch breaks the current logic to retrieve the base commit. We could add a workaround by setting --remote=<canonical remote name> (perhaps with origin as the default, with an override in a config somewhere), it just hasn't been a priority as of yet.
|
|
||
Comment 10•2 years ago
|
||
FWIW, I did get a failure during patch application, but that's also because for complicated reasons, it found the base revision to be 0000000000000000000000000000000000000000.
We could add a workaround by setting --remote=<canonical remote name>
We could scan the result of git remote for remotes with the url for mozilla-unified or mozilla-central.
|
||
Comment 11•2 years ago
•
|
||
(In reply to Mike Hommey [:glandium] from comment #10)
FWIW, I did get a failure during patch application, but that's also because for complicated reasons, it found the base revision to be 0000000000000000000000000000000000000000.
We could add a workaround by setting --remote=<canonical remote name>
We could scan the result of
git remotefor remotes with the url for mozilla-unified or mozilla-central.
There is something weird here, my push-to-try has been working very well since the beginning and today it is also starting to show me a 00000's base commit.
Edit:
I have multiple remotes, including origin and local-bare:
$ LC_ALL=C git remote show origin | head
* remote origin
Fetch URL: hg::https://hg.mozilla.org/mozilla-unified
Push URL: hg::https://hg.mozilla.org/mozilla-unified
HEAD branch: branches/default/tip
Remote branches:
bookmarks/aurora tracked
bookmarks/autoland tracked
bookmarks/beta tracked
bookmarks/central tracked
bookmarks/esr10 tracked
$ LC_ALL=C git remote show local-bare | head
* remote local-bare
Fetch URL: git://nas.home/m-c/
Push URL: git://nas.home/m-c/
HEAD branch: (unknown)
Remote branches:
android tracked
bookmarks/central tracked
bug1432719_speechd_errors tracked
bug1763819_leak_crashReporter tracked
bug1774062_ctrl+s_bomb tracked
I got the 000's base commit after I did branch from origin's bookmarks/central into my branch, then push it to local-bare. if I remove the pushed branch to this repo, I can get push-to-lando to work.
|
||
Updated•2 years ago
|
|
|
||
Comment 12•2 years ago
|
||
Is there any known workaround for this?
While it was only happening on some branches, it now fails on all of them.
I can no longer use --push-to-lando because of this.
FYI, I'm using git locally.
if I remove the pushed branch to this repo, I can get push-to-lando to work.
May be this, but I don't quite follow what it means.
Here is my local state.
$ LC_ALL=C git remote show origin | head
* remote origin
Fetch URL: hg::https://hg.mozilla.org/mozilla-central
Push URL: hg::ssh://hg.mozilla.org/mozilla-central
HEAD branch: branches/default/tip
Remote branches:
branches/COMM193a4_20100510_RELBRANCH/tip tracked
branches/COMM193a5_20100623_RELBRANCH/tip tracked
branches/COMM2000_20101116_RELBRANCH/tip tracked
branches/COMM2000_20110114_RELBRANCH/tip tracked
branches/COMM2000_20110314_RELBRANCH/tip tracked
|
|
Assignee | |
Comment 13•2 years ago
|
||
In bug 1866395 I added a function to return the canonical Firefox
remote and use it when determining the base ref. I overlooked
that this also needs to be used for get_branch_nodes to avoid
returning a set of branch nodes that doesn't begin from the same
base ref. Move the building of the --remotes arg to a separate
function and use it in get_branch_nodes.
|
||
Updated•2 years ago
|
|
||
Comment 14•2 years ago
|
||
|
|
Assignee | |
Comment 15•2 years ago
|
||
I just landed a patch that resolves at least one manifestation of this bug, however the fix is only for Git and I am certain we have some users on hg who have hit this. If this issue is still happening to you after rebasing on latest autoland, please let me know so we have a data point indicating this bug is still unresolved.
|
||
Comment 16•2 years ago
|
||
| bugherder | ||
|
|
||
Updated•2 years ago
|
|
|
||
Updated•2 years ago
|
|
|
Assignee | |
Updated•2 years ago
|
|
|
Assignee | |
Updated•2 years ago
|
|
||
Updated•2 years ago
|
|
||
Comment 18•2 years ago
|
||
|
|
Assignee | |
Comment 19•2 years ago
|
||
I found an issue with the Git patch parser after inspecting some patch data in Lando's DB, and a bugfix has just been deployed. Please continue using --push-to-lando and if you hit this issue again, comment here so I know the bug is still unresolved.
It still fails to submit multiple patches for me: bug 1865653.
|
|
Assignee | |
Comment 21•2 years ago
|
||
Just to clarify, you're still seeing this bug now when trying to create pushes based off latest m-c/autoland? Can you re-create the failed push off latest changes and paste:
- The output from your terminal.
- The failure email.
- Git log between the SHA of the "base commit" from the terminal and HEAD.
Thanks in advance.
Sure!
The output:
> ./mach try chooser --push-to-lando
Starting trychooser on http://127.0.0.1:5000
* Serving Flask app 'tryselect.selectors.chooser.app' (lazy loading)
* Environment: development
* Debug mode: off
* Running on http://127.0.0.1:5000 (Press CTRL+C to quit)
127.0.0.1 - - [26/Jan/2024 17:30:56] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [26/Jan/2024 17:30:56] "GET /static/style.css HTTP/1.1" 304 -
127.0.0.1 - - [26/Jan/2024 17:30:56] "GET /static/filter.js HTTP/1.1" 304 -
127.0.0.1 - - [26/Jan/2024 17:30:56] "GET /static/select.js HTTP/1.1" 304 -
127.0.0.1 - - [26/Jan/2024 17:31:09] "POST / HTTP/1.1" 200 -
127.0.0.1 - - [26/Jan/2024 17:31:09] "GET /static/style.css HTTP/1.1" 304 -
estimates: Runs 771 tasks (730 selected, 41 dependencies)
estimates: Total task duration 13 days, 8:41:36
estimates: In the top 8% of durations
estimates: Should take about 1:57:16 (Finished around 2024-01-26 19:28)
Auth0 token validated.
Using a412d893c38f632f34b44a9c5b3fd8cc30bb87be as the hg base commit.
Submitting stack of 4 nodes and the try commit.
Patches gathered for submission.
Submitting patches to Lando.
Lando try submission success, took 3.46 seconds. Landing job id: 76147.
The email:
Your request to land try push with tip commit 'Bug 1876754 - Add PermissionUtils::ParseSetParameters for webdriver' failed.
Reason:
Problem while applying patch in revision None:
hg error in cmd: hg import --no-commit -s 95 /tmp/tmpxh4lzyoo: applying /tmp/tmpxh4lzyoo
patching file testing/web-platform/meta/notifications/lang.https.html.ini
Hunk #1 FAILED at 0
1 out of 1 hunks FAILED -- saving rejects to file testing/web-platform/meta/notifications/lang.https.html.ini.rej
patching file testing/web-platform/tests/notifications/lang.https.html
Hunk #2 succeeded at 31 with fuzz 2 (offset 0 lines).
abort: patch failed to apply
The git log:
commit 9cc32654987da46a6a81d61f1536d62414d3f901 (HEAD -> bug1875257-2)
Author: Kagami Sascha Rosylight <saschanaz@outlook.com>
Date: Tue Jan 23 20:26:17 2024 +0100
Bug 1876754 - Add PermissionUtils::ParseSetParameters for webdriver
commit ee1bd5ca78c280aeabc849d1b849269f3222e34c (bug1875257)
Author: Kagami Sascha Rosylight <saschanaz@outlook.com>
Date: Thu Jan 18 19:21:11 2024 +0100
Bug 1875257 - Part 3: Remove dom.screenwakelock.testing r=#dom-core
`test_driver.set_permission()` should now just work with the marionette pref.
Differential Revision: https://phabricator.services.mozilla.com/D199211
commit c85bab526934c69158f5ab18040bed44a09622ab
Author: Kagami Sascha Rosylight <saschanaz@outlook.com>
Date: Thu Jan 18 15:57:02 2024 +0100
Bug 1875257 - Part 2: Adjust notification tests r=asuth,#dom-worker-reviewers
Fixing one test to set permission properly and another to not trigger external network request.
Differential Revision: https://phabricator.services.mozilla.com/D198957
commit 64fe430c94478fdde714b85ee200f1c258b4020a
Author: Kagami Sascha Rosylight <saschanaz@outlook.com>
Date: Wed Jan 17 18:56:20 2024 +0100
Bug 1875257 - Part 1: Generalize permission handling of storage-access to all permissions r=whimboo
Differential Revision: https://phabricator.services.mozilla.com/D198836
commit 89343e939e75da6cd348af565d7ee282623e210e (origin/bookmarks/central, main)
Author: Markus Stange <mstange.moz@gmail.com>
Date: Fri Jan 26 03:38:54 2024 +0000
|
||
Comment 24•2 years ago
|
||
Another instance:
Lando error:
Your request to land try push with tip commit 'Bug 1874040 - Simplify dir=auto implementation. r=smaug' failed.
Reason:
Problem while applying patch in revision None:
hg error in cmd: hg import --no-commit -s 95 /tmp/tmpoalm72ny: applying /tmp/tmpoalm72ny
patching file layout/reftests/bidi/dirAuto/1103348-1-ref.html
Hunk #1 FAILED at 0
1 out of 1 hunks FAILED -- saving rejects to file layout/reftests/bidi/dirAuto/1103348-1-ref.html.rej
patching file layout/reftests/bidi/dirAuto/1103348-1.html
Hunk #1 FAILED at 0
1 out of 1 hunks FAILED -- saving rejects to file layout/reftests/bidi/dirAuto/1103348-1.html.rej
abort: patch failed to apply
Mach try output:
Access token is expired.
1. On your computer or mobile device navigate to: https://auth.mozilla.auth0.com/activate?user_code=MDVP-FZGX
2. Enter the following code: MDVP-FZGX
Waiting....
Login successful.
Auth0 token validated.
Using e9309553f817fa164bde4c229fe12ae51207bed9 as the hg base commit.
Submitting stack of 3 nodes and the try commit.
Patches gathered for submission.
Submitting patches to Lando.
Lando try submission success, took 5.82 seconds. Landing job id: 77910.
Try push without lando: https://treeherder.mozilla.org/jobs?repo=try&revision=2ffb56fdd09dc2674e1900ba4e8840242191be6e
|
|
||
Comment 25•2 years ago
|
||
Pushing directly to try no longer works since friday, and --push-to-lando still fails for me on most patches.
Artifact builds enabled, pass --no-artifact to disable
estimates: Runs 17 tasks (16 selected, 1 dependencies)
estimates: Total task duration 0:20:02
estimates: Should take about 0:15:02 (Finished around 2024-03-04 02:40)
Access token is expired.
1. On your computer or mobile device navigate to: https://auth.mozilla.auth0.com/activate?user_code=XSZH-ZKNH
2. Enter the following code: XSZH-ZKNH
Waiting...........
Login successful.
Auth0 token validated.
Using 79823348e9cf740dbe5ae11eb654b1a85ea0ecbc as the hg base commit.
Submitting stack of 6 nodes and the try commit.
Patches gathered for submission.
Submitting patches to Lando.
Lando try submission success, took 26.21 seconds. Landing job id: 78840.
your request to land try push with tip commit 'Bug 1866814 - [devtools] Merge all DevTools JS Actors into a single DOM Process JS Actor.' failed.
Reason:
Problem while applying patch in revision None:
hg error in cmd: hg import --no-commit -s 95 /tmp/tmph59zw2b6: applying /tmp/tmph59zw2b6
patching file devtools/server/actors/watcher/WatcherRegistry.sys.mjs
Hunk #1 FAILED at 177
1 out of 1 hunks FAILED -- saving rejects to file devtools/server/actors/watcher/WatcherRegistry.sys.mjs.rej
abort: patch failed to apply
I was trying to push this patch queue https://phabricator.services.mozilla.com/D203409
Could it be that it doesn't pickup all the patches? Or wrong base revision?
It mentions 79823348e9cf740dbe5ae11eb654b1a85ea0ecbc which was the latest tip from m-c when I pushed.
|
|
Assignee | |
Comment 26•1 year ago
|
||
I just deployed a fix in bug 1893976 which I suspect will also resolve this bug. If you still have issues with --push-to-lando failing after submission, please add a comment here with details so I can debug further. I'll close this bug in a few days if nobody reports any breakage.
|
|
Assignee | |
Updated•1 year ago
|
Description
•