Closed Bug 1758797 Opened 2 years ago Closed 2 years ago

Profile.clone fails in copytree with shutil.Error: [('/tmp/tmpkju1y5w3.mozrunner/lock', '/tmp/tmptlkdegtt/lock', "[Errno 2] No such file or directory: '/tmp/tmpkju1y5w3.mozrunner/lock'")]

Categories

(Testing :: Mozbase, defect, P2)

Default
defect
Points:
1

Tracking

(firefox100 fixed)

RESOLVED FIXED
100 Branch
Tracking Status
firefox100 --- fixed

People

(Reporter: jdescottes, Assigned: jdescottes)

References

(Blocks 1 open bug)

Details

(Whiteboard: [bidi-m3-mvp] )

Attachments

(2 files)

When trying to use Profile.clone for a test in Bug 1750689, I had a permafailure on linux. The clone failed while trying to copy a lock symlink:

https://treeherder.mozilla.org/jobs?repo=try&revision=39eeb3ad14aa607db2f79cd6fba176d5b7365bec&selectedTaskRun=Bi8AfuMVTFuQceKBHUtJgA.0

[task 2022-03-08T23:18:57.122Z] 23:18:57     INFO - STDOUT: E           shutil.Error: [('/tmp/tmpkju1y5w3.mozrunner/lock', '/tmp/tmpdbvf_cdy/lock', "[Errno 2] No such file or directory: '/tmp/tmpkju1y5w3.mozrunner/lock'")]
[task 2022-03-08T23:18:57.122Z] 23:18:57     INFO - STDOUT: copy_function = <function copy2 at 0x7f2f3b08b158>
[task 2022-03-08T23:18:57.123Z] 23:18:57     INFO - STDOUT: dst        = '/tmp/tmpdbvf_cdy'
[task 2022-03-08T23:18:57.123Z] 23:18:57     INFO - STDOUT: dstname    = '/tmp/tmpdbvf_cdy/startupCache'
[task 2022-03-08T23:18:57.124Z] 23:18:57     INFO - STDOUT: errors     = [('/tmp/tmpkju1y5w3.mozrunner/lock',
[task 2022-03-08T23:18:57.124Z] 23:18:57     INFO -   '/tmp/tmpdbvf_cdy/lock',
[task 2022-03-08T23:18:57.124Z] 23:18:57     INFO -   "[Errno 2] No such file or directory: '/tmp/tmpkju1y5w3.mozrunner/lock'")]
[task 2022-03-08T23:18:57.124Z] 23:18:57     INFO - STDOUT: ignore     = None
[task 2022-03-08T23:18:57.125Z] 23:18:57     INFO - STDOUT: ignore_dangling_symlinks = False
[task 2022-03-08T23:18:57.126Z] 23:18:57     INFO - STDOUT: ignored_names = set()
[task 2022-03-08T23:18:57.126Z] 23:18:57     INFO - STDOUT: linkto     = '172.17.0.4:+17954'
[task 2022-03-08T23:18:57.127Z] 23:18:57     INFO - STDOUT: name       = 'startupCache'

I tried to expose ignore_dangling_symlinks as an argument of Profile.clone and forward it to copytree: https://treeherder.mozilla.org/jobs?repo=try&revision=90e124be8d63be936e2948da70bea394c4fa8f9c

It fixed the problem even though we went for a different solution in the end.

This bug is to propose to expose that argument, and potentially investigate why/when this lock symlink might appear and make the clone fail.

See Also: → 1750689

Strange. The lock file as actually written by Firefox to indicate that the profile is in use. It will be removed again when Firefox quits. What is the source of this operation exactly? Are you maybe trying to clone a profile that is currently in use by a wdspec test?

This error happens for the first test case of my test, meaning I haven't started Firefox before. And I am not starting Firefox with the test profile anyway, only with clones.

That would mean a previous test is not successfully clearing the lock file? The previous test is /webdriver/tests/take_element_screenshot/user_prompts.py

What is the source of this operation exactly?

The code is at https://hg.mozilla.org/try/rev/0101f06883cc0e110d64653a7c82b396be64dbe0
The clone call itself is:

profile_arg, profile_folder = firefox_options["args"]
self.profile = Profile.clone(profile_folder)

which happens in the constructor of the browser fixture used in my test.

I can try to delay the call to Profile.clone, in case it's a race and the lock file didn't have time to be cleaned up, but by the end of the test, several seconds passed, so surely that should be enough time for this lockfile to be cleaned up.

Important thing to note is that the lockfile is NOT present, there is only a dangling symlink to a lock file. Meaning the actual lock has been moved or deleted already. The failure happens because copytree tries to copy this lock symlink which points to nothing.
So maybe that's how the lock is cleaned up on Linux? it somehow keeps a dangling symlink which is usually fine but problematic here?

Oh I was actually wrong. The lock file that Firefox creates within a profile is called .parentlock. So not sure what this lock file is. Do you have a way to reproduce it locally and check with ls -la where this symlink actually points to?

At the end it might be indeed a good idea to ignore dangling symlinks by maybe even setting the argument to True by default, and not expose it at all to Profile.clone()?

Geoff, what do you think?

Flags: needinfo?(gbrown)

No way to reproduce locally, I don't have a linux dev env sadly.

But I reduced the test case at https://treeherder.mozilla.org/jobs?repo=try&revision=c54022aac6a9f4321882c98ea8c3930bd772cbb3

from mozprofile import Profile

def test_clone_profile(full_configuration):
    firefox_options = full_configuration["capabilities"]["moz:firefoxOptions"]
    profile_arg, profile_folder = firefox_options["args"]
    Profile.clone(profile_folder)
    assert True

and it still fails on Linux.

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #4)

At the end it might be indeed a good idea to ignore dangling symlinks by maybe even setting the argument to True by default, and not expose it at all to Profile.clone()?

Ignoring dangling symlinks by default in Profile.clone seems quite reasonable to me.

Flags: needinfo?(gbrown)
Assignee: nobody → jdescottes
Status: NEW → ASSIGNED
Pushed by jdescottes@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/84a9d5dceeba
Call copytree with ignore_dangling_symlinks=True from Profile.clone r=gbrown,whimboo
https://hg.mozilla.org/integration/autoland/rev/4713cc11a819
Use Profile.clone to create profile in wdspec test websocket_upgrade.py r=webdriver-reviewers,whimboo
Blocks: 1759189
Points: --- → 1
Whiteboard: [bidi-m3-mvp]
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → 100 Branch
Priority: -- → P2
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: