696634 - Moving profile folder between machines disables random add-ons (maybe a NTFS modification times bug)

Status: RESOLVED WORKSFORME

(Toolkit :: Add-ons Manager, defect)

Description

[Palmer Eldritch]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Build ID: 20110928134238

Steps to reproduce:

1. Exit firefox
2. Copy profile contents - with many addons and the profile folder being in C:\dropbox\FFprofile (custom location that is) - except Cache\, OfflineCache\, startupCache\, .autoreg && .parentlock (the last 2 files seem not to be deleted on exit on 7.0.1) - to a new machine - in my case from win7 pro SP1 to win XP home SP3.
2a. alternatively create a new profile - add some addons (for ideas see below) then exit and copy the profile folder in new machine
3. Launch FF in new machine

A bunch of addons will be tagged as incompatible and be disabled - updating addons fixes that (no updates found)

Rinse and repeat.


Actual results:

A bunch of addons will be tagged as incompatible and be disabled - updating addons fixes that (no updates found)

Very interesting feedback was given in FF answers thread here : https://support.mozilla.com/en-US/questions/874485?s=&r=1&as=s
--Apparently raring the profile folder then unraring it in destination squashes the bug - or using a FAT filesystem - but "If you switch off "high precision modification time" in Winrar, it stops working." Check the link in (no) answers for details

Please also check http://forums.mozillazine.org/viewtopic.php?f=9&t=2323559 for a list of addons that got disabled in my case - the list displays some consistency across FF versions etc but not 100% - using FF in windows XP, 7 basically

Please fix - big annoyance since FF 4 I think (but maybe just noticed it then) - and also maybe in the root of other bugs (a subtle one)


Expected results:

Have addons functioning out of the folder so to speak

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I would think this is just that the updated metadata from AMO isn't pulled in until the first check for updates, so all Firefox has to go off of is the addon's original metadata in the install.rdf file.

Comment 2

[Palmer Eldritch]

Please check the links provided - confirmation needed if it is an NTFS only issue

Comment 3

[Palmer Eldritch]

This rings a bell : [FF] checks the modification time of the add-on directory before checking install.rdf.

Comment 4

[s3m3n]

Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:8.0) Gecko/20100101 Firefox/8.0

Same thing here with Ubuntu 11.10 32bit on Ext3 partition. I can see that addons are not random, but always the same, in my case:
BugMeNot 2.2 false {987311C6-B504-4aa2-90BF-60CC49808D42}
Custom Buttons 0.0.5.5 false [e-mail]
Novell Moonlight 2.4.1 false [e-mail]
Polski slownik poprawnej pisowni 1.0.20110621 false [e-mail]

I'm syncing my firefox profiles over two machines, but never running both instances at the same time. Every sync above addons are disabled and after trying to find updates (but there are no updates) I see the "restart to enable" information.

Hope this will be solved some way, some time. Thanks

Comment 5

[Dave Townsend [:mossop]]

On startup we check the file modification times of extensions. If they have changed we reload their metadata from install.rdf, throwing away any update compatibility info that AMO may have given us previously. This will cause many add-ons to become incompatible until we next do an update check and sync metadata again. Everything you're saying matches what I'd expect from this and currently it is the expected behaviour so I'm closing as invalid. If you think there is something else go on then please reopen.

Comment 6

[Palmer Eldritch]

If this were the case simply moving the profile folder would not be a problem - it shouldn't !

Did you really read the links posted ?

All the info on the addons should be kept in the profile folder - when I first run FF and checks for incompatibility and decides that addon X is compatible it should carry this decision around *with* the profile folder. Which was the case with FF 3 btw. There is a bug somewhere in the whole process - as long as I use my profile folder it *from the same machine* works as it should - why moving the profile folder would make addons incompatible - which were already marked compatible ? Could you please explain this ?

Comment 7

[Dave Townsend [:mossop]]

(In reply to Palmer Eldritch from comment #6)
> If this were the case simply moving the profile folder would not be a
> problem - it shouldn't !

Depending on the filesystem and the OS moving files can alter the modification time of the file (particularly when moving across different filesystems).

If you attach extensions.sqlite from your profile folder from before the move and after you have used Firefox after the move and it has disabled some add-ons I could tell you for sure whether this was the case.

Comment 8

[Palmer Eldritch]

Thanks - I will be doing this ASAP - still in my case I move from NTFS (win7) to NTFS (XP)

Comment 9

[Palmer Eldritch]

Well here is extensions.sqlite from my profile folder from before the move : http://db.tt/bUqgmwjY and after I have used Firefox after the move and it has disabled some add-ons : http://db.tt/H9W26RmV

I did not actually move - I used dropbox to sync - but this (as detailed in the links posted) does not make a difference. Moreover DB relies on modification times to sync files (I guess) - so it is highly unlikely it messes with those

I am very curious to learn what is going on - this has been bugging me since FF 4 and it is really annoying, I am telling you. Maybe should be fixed even if expected behavior - as the profile folder should carry the info in it - it is the same version of FF and the exact same addons (version and all) - they should be marked compatible once and for all

Thank you very much indeed for your interest !

Comment 10

[Dave Townsend [:mossop]]

Looking at the two databases I can see that Firefox detected file modification time differences between the two runs for all the add-ons that got made incompatible unexpectedly. That basically matches what I expected to be happening. The question of course is where are these differences coming from.

Add-ons can be installed either packed (as a plain XPI file in the profile) or unpacked (as an extracted directory of files).

For almost all the packed extensions you have installed the file modification times differ by less than a second (0.84s is the largest difference). From this I would guess that Dropbox only records the file modification time at a resolution of one second. NTFS on the other hand stores it to the nanosecond. This is pretty much confirmed by the number of modification times the database lists with "000" at the end. The chances of this happening are of course 1 in 1000 yet your database shows many such cases.

For the unpacked extensions the modification times differ by much more, up to 28 days in a couple of cases. From this I would guess that Dropbox doesn't sync the file modification times of directories at all. This wouldn't surprise me, many filesystems don't even support file modification times on directories and most version control systems just ignore directories entirely, they focus on files and rebuild directories as needed. A simple way to verify this would be to open explorer and look in the extensions directory in detail view. Look at the last modified column and compare before and after for directories in the list.

There is one exception to this. The Places Maintenance add-on is installed as a packed XPI yet shows a modification time difference of 46.5 days, that one I can't explain.

Comment 11

[Dave Townsend [:mossop]]

(In reply to Dave Townsend (:Mossop) from comment #10)
> For almost all the packed extensions you have installed the file
> modification times differ by less than a second (0.84s is the largest
> difference). From this I would guess that Dropbox only records the file
> modification time at a resolution of one second. NTFS on the other hand
> stores it to the nanosecond. This is pretty much confirmed by the number of
> modification times the database lists with "000" at the end. The chances of
> this happening are of course 1 in 1000 yet your database shows many such
> cases.

Forgot to add that this doesn't surprise me either. It's pretty common to throw away modification time bits when all you care about is knowing if a file has changed. Chances are if it will change then a second or so resolution is plenty to let you know (especially if you then hash the file to check for actual changes as I bet Dropbox does). Different filesystems have different modification time resolutions, FAT for example only used to have a 2 second resolution! The extension manager is using all the accuracy it gets for no good reason except that we have access to it. It could probably stand to drop down to a single second resolution but that wouldn't really solve your problems here.

Comment 12

[Palmer Eldritch]

Ahh - thank you very much for your quite thorough reply. Much appreciated :)

Well - seems the modification time scheme for checking addon compatibility has its drawbacks (apparently on copy/move operations there are the same problems - so NTFS and other filesystems do not carry the mod time around down to their supported precision) - could be maybe replaced by a hash based scheme ?

Thanks again for your time and interest.

Comment 13

[Dave Townsend [:mossop]]

(In reply to Palmer Eldritch from comment #12)
> Ahh - thank you very much for your quite thorough reply. Much appreciated :)
> 
> Well - seems the modification time scheme for checking addon compatibility
> has its drawbacks (apparently on copy/move operations there are the same
> problems - so NTFS and other filesystems do not carry the mod time around
> down to their supported precision) - could be maybe replaced by a hash based
> scheme ?

The problem is that we do the check on every startup to see if any extensions have changed. Hashing an extension's files would be quite expensive and would slow down the startup of Firefox which is something we cannot do.

Comment 14

[Tony Mechelynck [:tonymec]]

(In reply to Dave Townsend (:Mossop) from comment #13)
[...]
> The problem is that we do the check on every startup to see if any
> extensions have changed. Hashing an extension's files would be quite
> expensive and would slow down the startup of Firefox which is something we
> cannot do.

OK. What about comparing the modification timestamps with only 2 seconds accuracy? This ought to be compatible with all known filesystems, and I suppose it would be good enough for the purpose of change detection, wouldn't it?

Comment 15

[Palmer Eldritch]

Another occurrence of the "bug" : I have a dual boot windows7/Ubuntu 11.10 system where the profile folder is on the windows partition (NTFS). When I use the profile from Ubuntu and then switch to windows a bunch of addons gets disabled - and vice versa.
The addons that get disabled are the ones that reside in *folders* inside extensions/.
Now this is more serious than the moving folder issue.
Appears the modification time scheme for checking addons compatibility has its limitations

Comment 16

[Palmer Eldritch]

As of firefox 21 (and a few versions back actually) this is completely eliminated ! Profiles are sync'ed perfectly add-on wise - no more disabled addons on start - accros computers and OSes. You have apparently revamped the whole mechanism (behind the scenes (I'd be curious for the details)) to the very better ! Thank you very much and you may close this (as FIXED) :)

Comment 17

[Tony Mechelynck [:tonymec]]

(In reply to Palmer Eldritch from comment #16)
> As of firefox 21 (and a few versions back actually) this is completely
> eliminated ! Profiles are sync'ed perfectly add-on wise - no more disabled
> addons on start - accros computers and OSes. You have apparently revamped
> the whole mechanism (behind the scenes (I'd be curious for the details)) to
> the very better ! Thank you very much and you may close this (as FIXED) :)

Resolving this as WORKSFORME because FIXED would imply that we know rexactly what fixed it when. One of the meanings of WORKSFORME is "The bug has disappeared but we aren't sure exactly when or why".