Closed Bug 1720302 Opened 4 years ago Closed 4 years ago

toolkit/themes/shared/icons/reload.svg in mozilla-central repository always shown as modified

Categories

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Product:
Developer Services
Component:
Mercurial: hg.mozilla.org
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: aryx, Assigned: zeid)

References

Details

(Whiteboard: [how to fix your local repo in comment 23])

Attachments

(2 files, 1 obsolete file)

Report file for hg issue6528
80 bytes, application/octet-stream
Details
image.png
256.42 KB, image/png
Details

jdescottes and other developers mentioned toolkit/themes/shared/icons/reload.svg getting flagged as modified in their local repositories.

This landed twice, the first one got backed out: https://hg.mozilla.org/integration/autoland/log/4a80defad64be68af0c49d360f044cd24fce24dd/toolkit/themes/shared/icons/reload.svg

<jcristau> i wonder if it's https://bz.mercurial-scm.org/show_bug.cgi?id=6528
<mak> that bug seems to imply pushing from a poisoned repo will poison the target repo... I wonder if this issue will now propagate with pulls

Issue has been reproduced with Mercurial 5.1.2, 5.8 and 5.8.1.

Zeid, please investigate.

Flags: needinfo?(zeid)

<marmoute> The issue is that 5.8 corrupt the storage, so once it is done, any other version will still see the issue. The corruption happens when a 5.8 hg send the changesets over the wire.

<marmoute> that corruption will be recoverable with a script scrubbing the repository to restore files to their proper state.

In a local test I commit a small change to the offending file (removed the trailing newline) and the problem went away. Backing out that change also left the repo in a good state. I don't know if that will resolve the problem after pushing but it sure would be easier to try than any maintenance on the repository itself.

Assignee: nobody → zeid
Flags: needinfo?(zeid)

Per jcristau in #developers, hg 5.8.1 should not introduce this corruption. It contains https://www.mercurial-scm.org/repo/hg/rev/411dc27fd9fd, which is tagged as 5.8.1. If the corruption has been introduced by an earlier bad version, then 5.8.1 will continue to witness it, i.e., it does not fix this corruption.

The corruption will have to be fixed directly on the remote repo, however folks can continue to use moz-phab to push to Phabricator by fixing up their local repo manually (i.e. to circumvent the error regarding uncommitted changes).

Another thing to note is that to avoid any additional similar corruptions, we should ensure that any changes that are pushed directly to hgmo from a client are done using a stable version of mercurial that is not affected by this issue (looks like <=5.7.1 and >=5.8.1 should do it).

Let me know if I can do anything to mitigate the annoyance here. I'm happy to re-land if backing out would help. Or add another newline/non-functional change to the file. This patch doesn't have user impact, its just moving a file from browser/ to toolkit/ and updating associated references

Workaround - I'm leaning on a central revision from yesterday to do my work from: hg up -C 5f345b

Blocks: 1720281
No longer blocks: 1720325

Here's the output from the remote repo confirming this bug is very likely https://bz.mercurial-scm.org/show_bug.cgi?id=6528:

rev linkrev nodeid       p1           p2
0  585313 2c5026382dae 000000000000 000000000000
1  585329 1fd153998403 000000000000 000000000000
2  585339 5e3e44b91515 1fd153998403 000000000000

The fix to scrub the repo and fix this issue is still a WIP by upstream. It could potentially have some impact on our replication once the fix is deployed.

Update: after chatting a little more with marmoute -- this output actually may not indicate the relation to the referenced bug. Will need to dig around in the morning.

Update2: it appears that autoland has the correct version, and mozilla-central has the corrupt version. The version from mozilla-central is above, and below is the autoland version, with the swapped parents. This confirms the bug is issue 6528.

rev linkrev nodeid       p1           p2
0  585327 2c5026382dae 000000000000 000000000000
1  585342 1fd153998403 000000000000 000000000000
2  585356 5e3e44b91515 000000000000 1fd153998403

Hi all, some updates about this issue.

While we wait for a fix from upstream so that we can fix the remote repo and any corruptions on local repos, you should be able to do the following to temporarily hide the problem on your local machine. This can be done with any revision just make sure your changes are committed:

hg up null --clean
hg up <rev>

The above commands will take some time to reload the entire directory from scratch, but it will update the timestamps in the directory state file to match those on the files on disk. It should fix the issue until the timestamp on the file in question (toolkit/themes/shared/icons/reload.svg) is updated again for any reason. This would cause it to deviate from what's stored in dirstate, which will trigger mercurial to lookup this file and mark it as changed due to issue 6528 .

The fix is anticipated next week.

Is this a bug of hg?

(In reply to Alexandru Ionescu (needinfo me) [:alexandrui] from comment #10)

Is this a bug of hg?

Yes, it's a hg issue, we are waiting for a hg server patch.

(In reply to Zeid Zabaneh [:zeid] from comment #9)

Hi all, some updates about this issue.

While we wait for a fix from upstream so that we can fix the remote repo and any corruptions on local repos, you should be able to do the following to temporarily hide the problem on your local machine. This can be done with any revision just make sure your changes are committed:

hg up null --clean
hg up <rev>

The above commands will take some time to reload the entire directory from scratch, but it will update the timestamps in the directory state file to match those on the files on disk. It should fix the issue until the timestamp on the file in question (toolkit/themes/shared/icons/reload.svg) is updated again for any reason. This would cause it to deviate from what's stored in dirstate, which will trigger mercurial to lookup this file and mark it as changed due to issue 6528 .

The fix is anticipated next week.

The workaround doesn't work for me. After trying it I can't even push to try or submit patches to phab.
Edit: I had to close pycharm.

Depends on: 1721228

Hi,

Is there any chance that the earlier version of hg (in my case 5.6.1)
regards a freshly fetched M-C tree to contain a local change or something due to the issues discussed here?
After I updated M-C tree this morning
I cannot apply local MQUEUE (I know it is not recommended.) any more.
It says
abort: local changes found
|hg diff| shows nothing.
Just for experimentation, I issue
|hg qnew temp| to take care of this phantom local change.
BUT, the next time I qpop the local change stored in |temp| patch, and they try to qpush this again,
hg complains
abort: local changes found
There SHOULD NOT BE a local change after all and so something is wrong with the repo.

Oh well, either this bug affects the repository subtly with older |hg| command, or I somehow messed up my repository by
running |hg pull -u| this morning.

Any ideas?

TIA
PS:

The version of M-C I have is this.
 hg log | head -100
changeset:   586141:a8a4dfcadce5
tag:         tip
fxtree:      central
user:        Marian-Vasile Laza <mlaza@mozilla.com>
date:        Tue Jul 20 16:33:08 2021 +0300
summary:     Backed out 11 changesets (bug 1721333) for causing crashtest/mochitest assertion failures on NativeObject.h. CLOSED TREE
...

(In reply to ISHIKAWA, Chiaki from comment #13)

Is there any chance that the earlier version of hg (in my case 5.6.1)

As per comment 2, once a repository has been corrupted by version 5.8 all versions of mercurial will see the issue. When you updated your repository you pulled the corrupt data into it.

(In reply to Dave Townsend [:mossop] from comment #14)

(In reply to ISHIKAWA, Chiaki from comment #13)

Is there any chance that the earlier version of hg (in my case 5.6.1)

As per comment 2, once a repository has been corrupted by version 5.8 all versions of mercurial will see the issue. When you updated your repository you pulled the corrupt data into it.

Thank you for the clarification.

It would be great to see an official guideline what a third-party developer/occasional patch producer can do to clean up the local repo very soon.

(In reply to ISHIKAWA, Chiaki from comment #15)

It would be great to see an official guideline what a third-party developer/occasional patch producer can do to clean up the local repo very soon.

Does comment 9 work for you for these purposes? It worked for me.

Whiteboard: [how to fix your local repo in comment 9]

(In reply to ISHIKAWA, Chiaki from comment #15)

It would be great to see an official guideline what a third-party developer/occasional patch producer can do to clean up the local repo very soon.

As a status update, I'm in talks with developers from upstream about releasing a script to find and fix the repo corruption. Once this script is implemented and tested I will share it here for anyone who unfortunately ran into this bug. Thanks for your patience!

https://bugzilla.mozilla.org/show_bug.cgi?id=1720302#c9 works for me sometimes and other times it does not (I can't recall the exact issues I was having at the time)

what I have been doing ( by advice from Mitch Henges ) is to create a single commit contain the svg file, before your changes, or rework your history to move this commit before. (This can be tricky in bookmarks that are already in progress but works well on new bookmarks)

then submit only your changes using moz-phab submit <start_rev> <end_rev>

I have at least once or twice accidentally committed that file using this approach, I recommend to check hg histedit or hg log every time to make sure there is a separate single commit for the hg error

I sure am learning a lot about mercurial because of this bug :)

Hi all,
It seems that depending on some environment factors, there are some cases when the above fix does not work. I've written a little python snippet that should be safe to run, and will fix the issue at least until the timestamp changes again for whatever reason. The main benefit is that this runs fast and does not require a full refresh of the repo. This should work on Linux and macOS. Just run this script from within your repo.

Paste the below into fix.py in the root directory of the repo. It requires mercurial to be installed via pip.

import os
from mercurial.pure.parsers import parse_dirstate

# Path of misbehaving file.
path = b"toolkit/themes/shared/icons/reload.svg"

if __name__ == "__main__":
    _map = dict()
    with open(".hg/dirstate", "rb") as f:
        # Read directory state file.
        dirstate = f.read()

        # Parse directory state into `_map`.
        parse_dirstate(_map, {}, dirstate)

        # Fetch the timestamp in directory state, then modify the file on disk to match.
        timestamp = _map[path][-1]
        os.utime(path.decode("utf-8"), (timestamp, timestamp))

Then do:

mozilla-central$ hg st
M toolkit/themes/shared/icons/reload.svg
mozilla-central$ python fix.py
mozilla-central$ hg st
mozilla-central$

Update: I added the above to a gist for ease of download and revising. Also, the updated script can be run from outside the repo and takes a repo path as an argument, to prevent unwanted clutter in the repo.

Do we have an ETA on a proper fix? For some reason I hit this constantly despite the above, possibly because I'm constantly switching between mozilla-central and a project branch, and it is really eating into my time.

I should clarify, I've been using the suggestion in comment 9, not yet tried that in comment 19.

Attached file fix.py (obsolete) (deleted) —
The content of attachment 9233747 [details] has been deleted for the following reason: See linked gist in comment 19.

Edit: You can now upgrade you Mercurial directly to 5.9.1+, run the repair command and avoid downgrading back to 5.8.1.

There is an official fix in the upstream Mercurial release 5.9rc1. To fix your repositories permanently you can do the following:

  1. Download/upgrade to Mercurial 5.9rc1 (or later versions for anyone coming across this in the future). For example if you use pip to install Mercurial, run pip install --upgrade --user mercurial==5.9rc1.
  2. (Optional): Download the attached "report" file and save it somewhere on disk. This is a pre-generated report that contains the list of affected revisions for mozilla-central/autoland and will cause the fix command to run instantaneously. You can run the command without the report, but it will need to scan the repository for affected revisions and will take a few minutes.
  3. cd into your corrupted repository and run:
    • without the report file: hg debug-repair-issue6528
    • with the report file: hg debug-repair-issue6528 --from-report <path/to/report-file>
  4. Downgrade Mercurial versions back to 5.8.1 to avoid existing issues with incompatible extensions: pip install --upgrade --user mercurial==5.8.1.

You should no longer see this bug affecting your repositories.

Whiteboard: [how to fix your local repo in comment 9] → [how to fix your local repo in comment 23]

(In reply to Masatoshi Kimura [:emk] from comment #25)

I had to reinstall the stable version of mercurial (5.8.1) after I followed the steps in comment #23 because some mercurial extensions complained about the too new version of mercurial:

** Unknown exception encountered with possibly-broken third-party extension "clang-format" (version N/A)
** which supports versions 5.3 of Mercurial.
** Please disable "clang-format" and try your action again.
** If that fixes the bug please report it to https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox%20Build%20System&component=Lint%20and%20Formatting
** Python 2.7.16 (v2.7.16:413a49145e, Mar  4 2019, 01:37:19) [MSC v.1500 64 bit (AMD64)]
** Mercurial Distributed SCM (version 5.9rc1)
** Extensions loaded: absorb, blackbox, clang-format, configwizard, evolve 10.3.2, firefoxtree, fsmonitor, histedit, js-format, mq, purge, push-to-try, rebase, show, strip
Traceback (most recent call last):
  File "d:\mozilla-build\python\Scripts\hg", line 43, in <module>
    dispatch.run()

Reinstall commands:

$ pip uninstall mercurial
$ pip install --upgrade --user mercurial

Added downgrade to the list of instructions in comment 23. Thanks for pointing that out, emk.

I've fixed the repository corruption server-side for mozilla-central, autoland and pine.

I could fix my local repository according to comment 23.
All is well now.
Thank you!

Given this is now repaired server-side and a repair script is published, I think we can call this bug resolved. Thanks to Zeid for investigating and writing the temporary fix.py script, and a big shoutout to the Octobus team for writing the upstream repair command.

Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED

I just got some weirdness that seems like it might be related to this. I'm hoping that I can get some help here to figure out what's going on and if I need to do something fix it or if my patch is okay to merge as-is.

I pushed a patch to Phabricator that locally does not report to touch reload.svg. On Phabricator, the file comes up as part of the patch, but says that no changes were actually made to it (which is confusing).

Perhaps this is related to testing I did on Oak? Oak was super out of date, and I had to merge in mozilla-central and push it back to Oak. Then I moved between the Oak and Central in a unified repo, copying (grafting) the patches I wrote on Central to Oak for testing. Then I came back to the patches on Central and submitted them to Phabricator. Like I said, it doesn't show reload.svg being touched at all, locally. But Phabricator seems... unsure.

This is the Phabricator patch.

Flags: needinfo?(zeid)

I am still having the issue despite running the repair command.

:bytesized -- I will check the Phabricator server to see if perhaps the repo on Phabricator is affected by this issue, and somehow only manifested now.

:alexandrui -- can you please paste the output of hg st and hg debugindex toolkit/themes/shared/icons/reload.svg?

Flags: needinfo?(zeid) → needinfo?(aionescu)

alexandru.ionescu@SVROLP01356:~/workspace/mozilla-central$ hg st
! image/test/crashtests/570451.png
! testing/web-platform/tests/css/css-ui/support/PTS/xc9n2c08.png
! testing/web-platform/tests/css/css-ui/support/PTS/xd0n2c08.png
! testing/web-platform/tests/css/css-ui/support/PTS/xd3n2c08.png
plus a very large list of ? including venv. Don't mind telling me how to get rid of that.

alexandru.ionescu@SVROLP01356:~/workspace/mozilla-central$ hg st | grep reload.svg
? toolkit/themes/shared/icons/reload.svg.orig

alexandru.ionescu@SVROLP01356:~/workspace/mozilla-central$ hg debugindex toolkit/themes/shared/icons/reload.svg
rev linkrev nodeid p1 p2
0 586905 2c5026382dae 000000000000 000000000000
1 586921 1fd153998403 000000000000 000000000000
2 586931 5e3e44b91515 000000000000 1fd153998403
3 586937 035b158b440e 5e3e44b91515 000000000000

Flags: needinfo?(aionescu)

:alexanderui looks like you may have some locally modified files, other than the file referenced in this bug, that is impacting your repo. This is a different issue unrelated to this bug. You'll want to make sure that any tracked, modified files are committed or reverted in order to move forward.

To remove untracked files from your repo, you can just delete those manually or try using the purge extension.

(In reply to Masatoshi Kimura [:emk] from comment #25)

Reinstall commands:

$ pip uninstall mercurial
$ pip install --upgrade --user mercurial

That didn't quite work, it uninstalled this:
c:\mozilla-build\python\lib\site-packages\hgdemandimport*
c:\mozilla-build\python\lib\site-packages\hgext3rd*
c:\mozilla-build\python\lib\site-packages\hgext*
c:\mozilla-build\python\lib\site-packages\mercurial-5.0.dist-info*
c:\mozilla-build\python\lib\site-packages\mercurial*
c:\mozilla-build\python\scripts\hg
c:\mozilla-build\python\scripts\hg.exe
and reinstalled this:
c:\users\xx\appdata\roaming\python\python27\site-packages\hgdemandimport*
c:\users\xx\appdata\roaming\python\python27\site-packages\hgext3rd*
c:\users\xx\appdata\roaming\python\python27\site-packages\hgext*
c:\users\xx\appdata\roaming\python\python27\site-packages\mercurial-5.9.1.dist-info*
c:\users\xx\appdata\roaming\python\python27\site-packages\mercurial*
c:\users\xx\appdata\roaming\python\python27\site-packages\python27.dll
c:\users\xx\appdata\roaming\python\scripts\hg
c:\users\xx\appdata\roaming\python\scripts\hg.exe

We moved it accordingly. Now we're at HG 5.9.1. Ran hg up null --clean followed by hg up -r <rev of M-C tip>. Elapsed time about 20 min (10 min per command). A fresh clone most likely wouldn't have been much slower.

Sorry, I should have obsoleted my comment. Comment #23 has been updated to add a re-install step.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: