1413452 - mozIJSSubScriptLoader caches files past updates and breaks things

Status: RESOLVED WONTFIX

(Core :: XPConnect, defect)

Description

[Dave Garrett]

I'm the developer of the Flagfox addon for Firefox. This is not about WebExtensions; that migration is pending and unrelated. (this is also unrelated to the issue I reported to AMO admins via email earlier today) This is about mozIJSSubScriptLoader.loadSubScript() being insane. Sorry, I don't have a reduced testcase for this as it's apparently hard to reproduce and Firefox fixed itself by chance when I hit it.

TL;DR: mozIJSSubScriptLoader.loadSubScript() of a JS file from an addon can cache it past updates and removals of the file, causing breakage even after the new version doesn't even use mozIJSSubScriptLoader or the loaded file anymore.

Long version: Some Flagfox users were getting a broken load resulting in an error icon instead of a flag. I traced the problem to mozIJSSubScriptLoader apparently caching files loaded using it from Flagfox, FOREVER. An old corruption check I wrote like a decade ago (literally; 10 years this month) triggered and said that the IPDB files' size didn't match what it said it should be in the metadata file (was only a sanity check, which apparently turned out to be useful). After update, Firefox was still sometimes loading the OLD VERSION of that file from the cache, and new Flagfox was choking on being fed old Flagfox's file. In Flagfox 5.1.28 & 5.1.29 I tried to hack around this using the same hack needed for the similar well known problem with properties files being cached past addon updates, and even tried passing ignoreCache:true to loadSubScriptWithOptions(). Nope, apparently some users were still hitting it. So, with the Flagfox 5.2.0 update I released yesterday, I dropped all use of mozIJSSubScriptLoader and got rid of both files I loaded with it. Guess what error I saw? Yeah, after update from AMO it worked fine, then I restarted Firefox and had no icon; only an error in the error console. It said it couldn't load the file from the URI to the first one of those two files, that no longer exists (defaultprefs.js). The file is gone and a sanity check grep of the new version finds no instances of the string "defaultprefs" anywhere. Firefox is barfing trying to load an old file that doesn't exist anymore for my bootstrap.js:startup(), for no reason. This is stupid.

The machine I stumbled upon the bug with had a pending Firefox update to 56.0.2, which completely fixed the issue, probably by wiping the relevant cache. I wish I backed up the profile the second it glitched, but I just tried the basic troubleshooting of turning it off and on again first, and the queued update fixed things.

I've got another user already reporting a similar problem, but they already have the update. Troubleshooting pending.

I may or may not have not seen this during testing because I used Firefox's load temporary addon feature from about:debugging (seeing as signing is needed for a normal load), which I suspect doesn't feel the need to cache the same way. I've already tried installing 5.1.29 from AMO into a new Firefox 56 profile, restarting Firefox, updating to 5.2.0 from AMO, and then restarting a couple more times. No problems trying this on Linux or Windows. I don't know what scares Firefox into kung fu death grip caching JS files, so I can't reproduce intentionally.

Anyone have a guess as to how I could force this stupid cache to clear? Flagfox 5.2.x is a bootstrapped extension, currently with an embedded WebExtension for preferences migration. (again, work on the full WebExtension port is ongoing) Which file in the profile is even holding this? addonStartup.json.lz4? Can I just delete this once on startup or something?

To state the obvious, this could affect Firefox hotfix updates and the like, so please don't try to hand-wave it away saying that the pending WebExtension-apocalypse is fix-enough, here.

Comment 1

[Dave Garrett]

Just to update the info here, I've posted Flagfox 5.2.1 to AMO with the only change being the addition of dummy files at the old paths, which should at least save me from the zombie cache rising from the dead to break startups long after I stopped using it at all.

Comment 2

[Dave Garrett]

1509550744330    addons.xpi    WARN    Exception running bootstrap method startup on {1018e4d6-728f-4b20-ad56-37578a4de76b}: TypeError: scope.IPDBmetadata is undefined (chrome://flagfox/content/ipdb.jsm:37:5) [...]

So, apparently someone reported their client still choking with dummy files, and according to the error message, it's because it didn't have the same symbols. (The word/symbol "scope" does not appear in the current version of ipdb.jsm at all.) Flagfox 5.2.2 posted to AMO with more lifelike dummy files.

This could be that Firefox is also hyper-aggressively caching JS passed to XPCOMUtils.defineLazyGetter, as well. All the new files are very clearly installed and being loaded; it's just the cache seemingly not willing to let this go.

The AMO editor that reviews/approves my updates thinks the underlying cause may be in one/all of: bug 1382546, bug 1356826, bug 1382762

I'm pretty sure this craziness only happens in recent versions of Firefox, so those bugs sound plausible.

Comment 3

[Dave Garrett]

(bug 1356826 is the only one with commits; others are related info)

Comment 4

[Andrew McCreight [:mccr8]]

Maybe Kris has some ideas about what to do.

Comment 5

[Kris Maglione [:kmag]]

It sounds like the startup cache isn't being flushed after one of your updates. Updating Firefox would fix that, since that forces a startup cache flush.

At this point, though, there's no way we'd uplift a fix for this to 56, and 57 does not support legacy extensions, so marking wontfix.

Comment 6

[Dave Garrett]

(In reply to Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) from comment #5)
> At this point, though, there's no way we'd uplift a fix for this to 56, and
> 57 does not support legacy extensions, so marking wontfix.

If you'd like to get an idea as to why addon developers are not thrilled with Mozilla right now, it'd be insta-WONTFIXing a serious Firefox bug without investigation because it ~probably~ will only hit addons running in the current main release branch that haven't switched to the 3rd new addon API in the past 4 years. Firefox ESR does still exist, you know, as do Thunderbird and SeaMonkey. Also, you're just guessing that this won't affect any of Mozilla's code. I see 408 hits for loadSubScript in mozilla-central via DXR, plenty of which are in test cases.

Could you at least respond to my question and tell me how to manually trigger an addon cache clear on install/update from JS? Or, please at least confirm which file the cache is stored in, so I can delete it when debugging. I've already got a few other cache hacks in the code to work around other known longstanding cache bugs that have never been fixed. You can see them here, in the official MDN doc on how to port an overlay extension to restartless:
https://developer.mozilla.org/en-US/Add-ons/How_to_convert_an_overlay_extension_to_restartless

I'm the original author of that, by the way; Mozilla didn't actually do enough real-world testing and write sufficient usable documentation, so MDN asked me to adapt a blog post of mine to be posted there.

A little more information and a little less instant dismissal would go a long way towards not demoralizing the Mozilla community.

Comment 7

[Kris Maglione [:kmag]]

I'm not guessing. The startup cache is cleared at startup any time the build ID changes, which means that stale cache entries don't affect built-in code. WebExtensions do not use the startup cache. If this affects ESR, that may be worth fixing, but you suggested that it was a recent regression.

Thunderbird and Seamonkey are officially unsupported by Mozilla. If the people working on those projects want issues fixed that affect them specifically, it's up to them to fix them.


If you want a workaround, you can launch Firefox with the -purgecaches flag, or dispatch a startupcache-invalidate observer notification.

Comment 8

[Dave Garrett]

(In reply to Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) from comment #7)
> If this affects ESR, that may be
> worth fixing, but you suggested that it was a recent regression.

Fair point, but I don't know enough about this to say for certain, nor do I know what things have been backported to ESR updates, or how the various test cases could be affected. As I said, I have no steps to reproduce this, so I'm posting here with way less information than I would ideally like.

> Thunderbird and Seamonkey are officially unsupported by Mozilla. If the
> people working on those projects want issues fixed that affect them
> specifically, it's up to them to fix them.

Sigh. It is a Mozilla XPCOM interface, though. It's not particularly friendly to pawn off Mozilla bugs onto Thunderbird and SeaMonkey developers. (I personally use Thunderbird; I couldn't care less about SeaMonkey, though, I do maintain support for it in addons for some strange reason)

> If you want a workaround, you can launch Firefox with the -purgecaches flag,
> or dispatch a startupcache-invalidate observer notification.

Ah, thank you. Those two bits of info are exactly what I was looking for. I don't intend to attempt any further workarounds in published code, though, as I ~think~ I've got things dealt with for my case, now.