Open Bug 1543752 Opened 5 months ago Updated 9 days ago

Stop reading user.js from the profile to modify default prefs unless prefs.js opts in to doing so

Categories

(Core :: Preferences: Backend, defect, P5)

defect

Tracking

()

Tracking Status
firefox68 --- affected

People

(Reporter: Gijs, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: main-thread-io, perf, Whiteboard: [fxperf:p2] [fxperfsize:S])

I've never fully understood the point of having this file. It overrides the defaults, from the profile directory. But people abuse it and break stuff (cf. bug 1534282) because they don't understand it. There's nothing that you can't do by modifying things using the application dir's set of default pref modifying files, or through the enterprise policy work, and in general modifying the default values for prefs (instead of the user value) is likely to lead to hard-to-understand/debug issues, like in the aforementioned bug, but also because our internal code might use "does this pref have a user value" as a shortcut to "are we doing a non-default thing". Modifying the update channel via the default prefs is also likely to cause "fun" issues with incremental updates, I expect.

Orthogonally, checking for this file needlessly causes additional IO early on startup.

I propose we just stop supporting it.

Dave, Nick, Mike, am I missing a reason we should keep it? Who (else) would need to sign off on removing it?

If there are compelling reasons to keep support for it, I'd like to suggest we only read it if there's a user pref in prefs.js that indicates it ought to be read.

Anecdotally, the reason people use it is because it gives them a really easy way to maintain preferences when they reinstall/move/change Firefox.

I think we actually have telemetry on usage.

As far as only reading it a pref is set, that seems like a good compromise to removing it. Require someone to explicitly choose to use it.

Meant to needinfo so just doing that now. Thanks to Mike for weighing in already.

Flags: needinfo?(n.nethercote)
Flags: needinfo?(dtownsend)

(In reply to Mike Kaply [:mkaply] from comment #1)

Anecdotally, the reason people use it is because it gives them a really easy way to maintain preferences when they reinstall/move/change Firefox.

I think we actually have telemetry on usage.

Can you point me to that telemetry? I couldn't find it from a quick look...

Flags: needinfo?(mozilla)

https://searchfox.org/mozilla-central/source/modules/libpref/Preferences.cpp#4032

Telemetry::ScalarSet(Telemetry::ScalarID::PREFERENCES_READ_USER_JS, true);

I've never looked at it in telemetry, I just knew it was set.

Flags: needinfo?(mozilla)

(In reply to Mike Kaply [:mkaply] from comment #4)

https://searchfox.org/mozilla-central/source/modules/libpref/Preferences.cpp#4032

Telemetry::ScalarSet(Telemetry::ScalarID::PREFERENCES_READ_USER_JS, true);

I've never looked at it in telemetry, I just knew it was set.

Ah, whereas I looked at telemetry.m.o, and it expired in 62 so it's not in there for current nightly...

Anyway, 2.5% of pings in nightly 61, 2.7% of pings in beta 61. Of course, that still doesn't tell us who put it there, what's in it, and if users actually want it...

The main difference between user.js and prefs.js, IIRC, is that the contents of user.js are sticky, which prefs in prefs.js aren't. So if user.js contains foo.bar set to qux, and Firefox's default is baz, the user gets qux. When later Firefox's default switch to qux, the user gets qux. When much later Firefox's default changes to hoge or back to baz, the user still gets qux.

Without user.js, in the latter case, the user gets the new Firefox default.

(In reply to Mike Hommey [:glandium] from comment #6)

The main difference between user.js and prefs.js, IIRC, is that the contents of user.js are sticky, which prefs in prefs.js aren't. So if user.js contains foo.bar set to qux, and Firefox's default is baz, the user gets qux. When later Firefox's default switch to qux, the user gets qux. When much later Firefox's default changes to hoge or back to baz, the user still gets qux.

Yes, I'm aware - but to me this doesn't seem like a good reason for this thing to exist. Like, I get that that's how it works, but I don't think that's a net upside, when people can easily change the same pref in the unlikely event that the default really does change. When we change the default we expect the default to actually be different. We change things all the time, there aren't always prefs, and the number of pref defaults that have changed multiple times in this way (where old values are still supported) can surely be counted on one hand, even if we go back 10 years (which includes 2 major UI refreshes that garnered way more complaints than a few pref defaults changing). The internet's guidance on doing this seems to not discourage the method of "change all the settings you want, then just copy prefs.js to user.js" which is a recipe for disaster (all migrations will run on every startup, CustomizableUI will be permanently stuck in some previous state so subsequent add-on installs won't be customizable, it's likely even newly introduced Firefox UI won't get dealt with correctly, etc. etc.).

Once things break it's a nightmare to figure out /why/ things are broken, because generally they just copied some instructions off a random webpage 3 years ago and forgot all about doing that, they just wonder why a particular Firefox widget is fubared and either switch to Chrome/Safari/Edge/whatever or cause us / sumo / mozillazine to spend ages debugging for a problem that, fundamentally, shouldn't exist.

Anyway, I'd be highly surprised if 3% of beta pings are from users who actually want this thing... seems more likely that it's abused by other apps (AV or otherwise).

(In reply to :Gijs (he/him) from comment #8)

Anyway, I'd be highly surprised if 3% of beta pings are from users who actually want this thing... seems more likely that it's abused by other apps (AV or otherwise).

Seems likely to be search hijacking, given that the search service uses the default branch to read preferences that the user isn't expected to tweak, eg. defaultenginename (the original default engine that we use as a fallback if all else fails), geoSpecificDefaults.url (the url of the absearch server), ...

Seems likely to be search hijacking, given that the search service uses the default branch to read preferences that the user isn't expected to tweak, eg. defaultenginename (the original default engine that we use as a fallback if all else fails), geoSpecificDefaults.url (the url of the absearch server),

defaultenginename is only read if there is a distribution. We've never seen this used for hijacking.

user.js has been around for over 15 years and people have known about it for that long. There are articles everywhere about it. ghacks posts custom user.js files. There's tooling that creates user.js files - http://configfox.sourceforge.net/.

Your assumption that it is only used for nefarious purposes is misplaced. Just a couple weeks ago I was at a conference for Mac administrators and the subject of user.js came up. Lots of people use it for regular purposes.

Just because you don't see a use for user.js doesn't mean other people don't. Firefox is a very configurable browser. We shouldn't be surprised when people configure it.

I also think it's wise to be conservative here. We don't have a clear idea what effects removing this would have.

Flags: needinfo?(n.nethercote)

(In reply to Mike Kaply [:mkaply] from comment #10)

Seems likely to be search hijacking, given that the search service uses the default branch to read preferences that the user isn't expected to tweak, eg. defaultenginename (the original default engine that we use as a fallback if all else fails), geoSpecificDefaults.url (the url of the absearch server),

defaultenginename is only read if there is a distribution. We've never seen this used for hijacking.

user.js has been around for over 15 years and people have known about it for that long. There are articles everywhere about it.

News to me, but OK. about:config and prefs.js, yes, but user.js ?

ghacks posts custom user.js files.

  1. ghacks are very niche. This isn't anywhere near sufficient to explain 3% of pings.
  2. They seem to have stopped doing that in 2017.

There's tooling that creates user.js files - http://configfox.sourceforge.net/.

That (based on the changelog) hasn't been updated since 2015, since before photon and the last supported ESR.

Your assumption that it is only used for nefarious purposes is misplaced.

My main assumption is that 3% of users shouldn't want/need this thing, and 97% of users shouldn't be paying startup costs for it.

Just a couple weeks ago I was at a conference for Mac administrators and the subject of user.js came up. Lots of people use it for regular purposes.

Which are what? I'd expect admins to use the other default pref locations for changing defaults, as user.js is per-profile. What value does user.js provide to them / why can't they use the app-wide location?

Just because you don't see a use for user.js doesn't mean other people don't.

Come on, don't patronize me. I've been around here long enough to know this...

Firefox is a very configurable browser. We shouldn't be surprised when people configure it.

This is a strawman. I'm not objecting or surprised people configure Firefox. I'm surprised that they would use this file to do it.

It's reasonable to be surprised when 30-300x more people apparently create a config file in a hard-to-find folder for which there is no UI, no documentation (only on "here's a list of reasons things might not work properly" pages on SUMO), no prompting, than use some browser features like feed support for which we did have UI, the removal of which was described by some as the end of the bloody world. Same ratio with master password use, (0.11% of beta pings). 3-4x more users than people use any of the URL bar page action buttons (this includes bookmark, pocket, anything they customized there themselves, add-ons in the URL bar itself).

The stats here are surprising -- and also uncertain, because the probe only records pings where the file exists, not pings where it doesn't, so I'm forced to compare with other pings (I used GC_MS to get the 3% number).

In any case, it seems your argument is there is a usecase for this file. I'm aware of the one (other) Mike described in comment 6 - is there something else I'm missing? Besides the reasons Mike cited, and besides "setting the machine-wide defaults" for which you could use the app locations, why would you want to change the default value on a per-profile basis instead of changing prefs.js / about:config ?

Flags: needinfo?(mozilla)

I don't think I have much useful to say here since I don't understand the use cases.

Flags: needinfo?(dtownsend)

why would you want to change the default value on a per-profile basis instead of changing prefs.js / about:config ?

Well the advantage here is that you can have a file that you keep around and just drop into a profile directory and Firefox doesn't mess with it. That's what I've heard anecdotally.

I'm fine with switching this to a pref that's read to then read user.js. That makes sure that only people that want to use it use it.

I think the only problem there would be that if some preferences had already been user set to default values that then get set by user.js, there might be an issue.

We could also do some user surveying (similar to what was done around userContent.css/userChrome.css) to figure out if there is a usecase we're missing.

And I agree with you 100%. People complain loudly before removing features and then when we do remove them, there is just a whimper.

Maybe the secret is to start with only a browser and a URL bar and add features when they requested. Then we're never stuck with features people don't want :)

Flags: needinfo?(mozilla)

To give you another reason why some people use the file user.js: I can't speak for support in other languages but in German-speaking Firefox support we often recommend to use user.js instead of about:config as best practice and Gijs wrote the main reason, probably without knowing that he wrote about a good reason to use user.js: :P

(In reply to :Gijs (he/him) from comment #8)

Once things break it's a nightmare to figure out /why/ things are broken, because generally they just copied some instructions off a random webpage 3 years ago and forgot all about doing that, they just wonder why a particular Firefox widget is fubared and either switch to Chrome/Safari/Edge/whatever or cause us / sumo / mozillazine to spend ages debugging for a problem that, fundamentally, shouldn't exist.

That's exactly what happens if people apply changes via about:config: they forget that they changed things and what they changed. In the user.js file people can add notes what the options do. So people a) find all their changed options (which is, by the way, much more difficult with the new about:config!) and b) they can have notes about all their changes.

Of course, with user.js people still can forget that they changed things and what the options do. But wihthout user.js people will forget (almost) for sure.

Bonus point: It's very easy to apply changes to more than one system. Since about:config does not have an export/import feature it's a really nice feature for people who prefer to have the same settings on all their systems.

I hope this comment is helpful.

(In reply to Sören Hentzschel from comment #15)

Bonus point: It's very easy to apply changes to more than one system. Since about:config does not have an export/import feature it's a really nice feature for people who prefer to have the same settings on all their systems.

The solution for import/export is generally considered to be "use sync"; there's a separate enhancement request for export/import for prefs, I'm pretty sure. But IMHO we have already lost if people routinely have to resort to setting prefs in about:config or similar.

I hope this comment is helpful.

It is - it helps me understand why the German support forum might suggest this to people, but for the sake of the German support forum I hope you're not having to tell 3% of our users this, as we should then probably hire fulltime people to work German support given the volume of users that implies.

What do you tell people to change in user.js in this way anyway ? Is it really routine that people have to change prefs that aren't in the UI? Which ones? That in itself is pretty concerning tbh...

Anyway, the problem is that none of our support or engineering system is structured for detecting that this has happened (apart from, apparently, the German support forum). Some prefs, as already said, will break things if users change the default value. about:config won't say that prefs are modified if modified through user.js . about:support doesn't tell you which prefs are set, either, just whether the file exists (also, I just found bug 1544573 which makes this even more fun).

I assume that the support forum can deal with having to set a pref in prefs.js in order to enable user.js, or using the app dir version instead ? (ni for this)

Longterm, I'd really like to evaluate whether we can remove support for this file entirely, because it just fundamentally doesn't really make sense to have so many different files that all control the same thing, but it probably requires figuring out why so many people use it, which we don't have cycles to do. Nor is it really obvious how we'd go about doing so: if we think a substantial portion of people aren't aware they've done this, just doing a survey "why do you have this file" is unlikely to be enlightening; we could try doing telemetry on what prefs get set, but we'd probably have to have some kind of strict list of prefs we allow ourselves to send back to avoid passing back user data, which again might not get us the data we need.

(In reply to Mike Kaply [:mkaply] from comment #14)

I'm fine with switching this to a pref that's read to then read user.js. That makes sure that only people that want to use it use it.

I think the only problem there would be that if some preferences had already been user set to default values that then get set by user.js, there might be an issue.

I'm confident this works because it's more or less what Normandy does today. You can write to the default pref branch and it'll just overwrite the previous value. We read prefs.js and user.js one (immediately) after the other today based on what I'm seeing in the performance profile IO markers, and given one gets read into the default branch and one into the user branch, that should continue to Just Work.

Flags: needinfo?(soeren.hentzschel)
Summary: Stop reading user.js from the profile to modify default prefs → Stop reading user.js from the profile to modify default prefs unless prefs.js opts in to doing so

Making this opt-in based on a prefs.js pref should be straightforward in theory, and I haven't heard objections so far, so marking this up for that purpose.

Whiteboard: [fxperf] → [fxperf:p2] [fxperfsize:S]

(In reply to :Gijs (he/him) from comment #16)

What do you tell people to change in user.js in this way anyway ? Is it really routine that people have to change prefs that aren't in the UI? Which ones? That in itself is pretty concerning tbh...

Totally different because people are so different. ;-) browser.tabs.loadBookmarksInTabs is one example for a change that is "frequently" asked for (not frequent at all in relation to the total user base but it's more frequent than other requests). There is no visible setting so we have to tell the users that they have to use about:config or user.js. browser.tabs.tabMinWidth is another example.

But it's not always that we ask people to change a hidden setting. People report problems because they forgot that they changed something in the past. Popular examples are javascript.enabled and privacy.resistFingerprinting. People read articles on websites which recommend settings like privacy.resistFingerprinting because of some privacy improvements but these articles don't say anything about the side effects. People change these settings and a few months later they need help because something is broken like the installation of add-ons on addons.mozilla.org. In such situations we tell the users about the advantages of user.js (have an overview about all changed hidden settings and the ability to make comments).

(In reply to :Gijs (he/him) from comment #16)

I assume that the support forum can deal with having to set a pref in prefs.js in order to enable user.js, or using the app dir version instead ? (ni for this)

Yes, I don't see a big problem with that.

Flags: needinfo?(soeren.hentzschel)

The priority flag is not set for this bug.
:njn, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(n.nethercote)
Flags: needinfo?(n.nethercote)
Priority: -- → P5

I thought we had fixed this already. Here is a startup profile where opening the user.js (non-existent) file is clearly visible: https://perfht.ml/2ZpWcvw

(In reply to Florian Quèze [:florian] from comment #20)

I thought we had fixed this already. Here is a startup profile where opening the user.js (non-existent) file is clearly visible: https://perfht.ml/2ZpWcvw

180ms to open a non-existing file seems ridiculous, especially compared to other I/O happening earlier. I don't know what the profiler is supposed to be reporting there, but it seems fishy that there's only 1 sample for 180ms spent there. On a profile with an interval of 1ms, I'd expect 180. At this point, I'm starting to doubt what the profiler is reporting for I/O, and I'd be curious to know what e.g. Xperf would say happened during the same interval of time.

Assignee: nobody → mcheang

(In reply to Mike Hommey [:glandium] from comment #21)

180ms to open a non-existing file seems ridiculous, especially compared to other I/O happening earlier.

I/O time is random. It's typical that some I/O calls will take <1ms and the next one will block for more than 100ms. We even sometimes see this when opening several times the same file. Eg. in this same startup profile https://perfht.ml/2T6mH6O , opening C:\Program Files\Firefox Nightly\fonts\TwemojiMozilla.ttf takes 50ms on the parent process, and a few seconds later 606ms on the WebExtensions process.

I don't know what the profiler is supposed to be reporting there, but it seems fishy that there's only 1 sample for 180ms spent there.

Significant gaps in the sampling usually indicate that the machine is CPU starved. Which is pretty common on low end hardware.

On a profile with an interval of 1ms, I'd expect 180.

On Windows the minimum sampling rate currently supported is 2ms. So even when the sampling is configured at a 1ms interval, you would get at most 90 samples during a 180ms period on Windows. Even if the machine had plenty of spare CPU cycles available.

At this point, I'm starting to doubt what the profiler is reporting for I/O, and I'd be curious to know what e.g. Xperf would say happened during the same interval of time.

I/O interposition causes significant overhead on Windows, but that overhead is after the I/O markers, when trying to find the filename associated with the file handle. This overhead is because the HandletoFilename function itself triggers I/O. Which again takes a random potentially long time. In the same profile, we can see some of this overhead here: https://perfht.ml/2OCA7J8

(In reply to Florian Quèze [:florian] from comment #22)

(In reply to Mike Hommey [:glandium] from comment #21)

180ms to open a non-existing file seems ridiculous, especially compared to other I/O happening earlier.

I/O time is random. It's typical that some I/O calls will take <1ms and the next one will block for more than 100ms. We even sometimes see this when opening several times the same file. Eg. in this same startup profile https://perfht.ml/2T6mH6O , opening C:\Program Files\Firefox Nightly\fonts\TwemojiMozilla.ttf takes 50ms on the parent process, and a few seconds later 606ms on the WebExtensions process.

Is that your conclusion from observing profiles from the Gecko profiler, or with something else like Xperf? If there is something fishy in the Gecko profiler, you have some sort of case of confirmation bias occurring.

The mozprofile python library, used in much of our testing, creates profiles and uses user.js to override default prefs. I don't know who else is using it at the moment, but our team (Iris) relies on it. The goal is to put a profile into a certain state before launching Firefox. If we are required to change a preference first before this file is recognized, I imagine that would break us pretty badly.

If I'm missing something, please let me know.

(In reply to Matt Wobensmith [:mwobensmith][:matt:] from comment #24)

The mozprofile python library, used in much of our testing, creates profiles and uses user.js to override default prefs. I don't know who else is using it at the moment, but our team (Iris) relies on it. The goal is to put a profile into a certain state before launching Firefox. If we are required to change a preference first before this file is recognized, I imagine that would break us pretty badly.

You could write a prefs.js file with the pref to tell it "read the user.js file" as well, right? (In fact, you could probably just put all the prefs in prefs.js directly if it's a new profile...)

(In reply to :Gijs (he/him) from comment #25)

You could write a prefs.js file with the pref to tell it "read the user.js file" as well, right? (In fact, you could probably just put all the prefs in prefs.js directly if it's a new profile...)

True, thanks for that. We could make a workaround for our project, but it does affect other consumers of mozprofile. I suppose the maintainers of that library could account for this and do what you suggest automatically, thus enabling current use cases.

Assignee: mcheang → nobody

ghacks posts custom user.js files.
...
They seem to have stopped doing that in 2017

It's not true.

(In reply to :Gijs (he/him) from comment #12)

(In reply to Mike Kaply [:mkaply] from comment #10)

ghacks posts custom user.js files.

  1. ghacks are very niche. This isn't anywhere near sufficient to explain 3% of pings.
  2. They seem to have stopped doing that in 2017.

this was already mentioned, but i want to mention again that the ghacks config was moved to github long ago and has been a very active project ever since - how many people use it, i don't know, but i suspect the number is higher than you imagine - the ghacks repo also includes an update script to fetch the newest user.js

i've written several technical articles regarding Firefox and the 2 general config articles, both of which rely heavily on the ghacks user.js, are by far the most popular articles on my website for years (and they are kept updated)

Your assumption that it is only used for nefarious purposes is misplaced.

My main assumption is that 3% of users shouldn't want/need this thing, and 97% of users shouldn't be paying startup costs for it.

97% of users aren't using a user.js, so how is there a startup cost for them?

Just a couple weeks ago I was at a conference for Mac administrators and the subject of user.js came up. Lots of people use it for regular purposes.

Which are what? I'd expect admins to use the other default pref locations for changing defaults, as user.js is per-profile. What value does user.js provide to them / why can't they use the app-wide location?

when multiple users are sharing an installation?

It's reasonable to be surprised when 30-300x more people apparently create a config file in a hard-to-find folder for which there is no UI, no documentation ...

there's docs all over, including on mozilla.org

... why would you want to change the default value on a per-profile basis instead of changing prefs.js / about:config ?

a, so you guys can't change it back (no offense) and b, simplicity - admins probably aren't using this file as you imagine, but lots of general home users probably are and the user.js makes pref management easy, portable, and it survives changes to prefs.js

and using about:config is a horrible way to incorporate custom prefs for those that know what they're doing, and perhaps in general - the only reason i ever suggest using about:config is to test something and then revert it back

i suggest taking a look at the [https://github.com/ghacksuserjs/ghacks-user.js](ghacks repo) which i suspect is the most popular distributor of a custom user.js, at least in the U.S.

as for the this ticket, as a user i'm semi-ok with having a pref to tell the browser to read user.js, but i think you're gonna anger a lot of people who have a user.js for good reason (privacy, security) and they find out at some point down the road that it's being ignored

frankly i think the suggestion to ignore user.js is a monumentally bad idea

I totally concur with @bugzee, esp. with:

"as for the this ticket, as a user i'm semi-ok with having a pref to tell the browser to read user.js, but i think you're gonna anger a lot of people who have a user.js for good reason (privacy, security) and they find out at some point down the road that it's being ignored
frankly i think the suggestion to ignore user.js is a monumentally bad idea"

Configuration of FF is its strongest point. IMO, making that harder or impossible to do (which has already happened quite a bit lately) is shooting yourselves in the foot, IMO.

The solution for import/export is generally considered to be "use sync"; there's a separate enhancement request for export/import for prefs, I'm pretty sure.

Until then, there's no way to export the modified preferences. There's not even a way to list them, and in any case most of them are changed automatically by the browser. Do I want to export devtools.debugger.pending-selected-location, datareporting.policy.dataSubmissionPolicyNotifiedTime, or browser.urlbar.placeholderName? I have no idea what they are. But I know what's in my user.js file.

And of course, there will be people that won't want (or be able to) use Sync. Or say that I use Sync for my changed preferences. What do I do if one of them causes a problem a couple of years down the line? How do I even know it's there?

But IMHO we have already lost if people routinely have to resort to setting prefs in about:config or similar.

Perhaps we should remove support for about:config, then? Just so people don't break their profiles when digging through it? With user.js I only have to keep a file in sync across all my PCs. Should I make a list of the settings I need to toggle in the UI? Are all of those preferences exposed there?

(In reply to bugzee from comment #28)

My main assumption is that 3% of users shouldn't want/need this thing, and 97% of users shouldn't be paying startup costs for it.

97% of users aren't using a user.js, so how is there a startup cost for them?

The cost is touching the disk to check if the user.js file exists on the disk.

as for the this ticket, as a user i'm semi-ok with having a pref to tell the browser to read user.js, but i think you're gonna anger a lot of people who have a user.js for good reason (privacy, security) and they find out at some point down the road that it's being ignored

We could land the changes for this bug in two steps, with one release checking for user.js and creating the pref if a user.js file was found, and the future releases after that only touching the disk for user.js if the pref is set. This is how we handled the same change for userContent.css in bug 1541233 with bug 1550157 that landed first.

Please stop this.

I have been using netscape navigator / firefox for decades and user.js has certainly been around > 15 years.

I am really getting tired of doing enterprise and personal support of firefox and dealing with changes that completely break things because someone makes a claim that startup time is critical, or doesn't see a use-case for themselves:
(per Kaply's comment that you don't hear users loudly complaining post-change, perhaps it's because people are giving up and leaving. My frustration level with all the recent breakages with profiles and Policies not supporting things like "defaultPrefs()" and my feeling that all the stuff that worked great for the past decade (autoconfig/mozilla.cfg/defaultPref/lockPref/...) is going to be replaced with something less functional is very high)

$ strace -f -e trace=stat -c firefox -new-instance -profilemanager
% time seconds usecs/call calls errors syscall


100.00 0.011346 5 2205 1548 stat


100.00 0.011346 2205 1548 total
(IGNORING the 40 subprocesses spun off during that interval that i'm sure are consuming a lot more wallclock-time)

Please tell me how ONE stat() for a missing file is gonna make any real difference here. Same for the removal of processing user{Chrome|Content}.css.

prefs.js is NOT an answer here, as the entire point behind user.js is to do something before and independent of prefs.js. prefs.js is volatile, user.js is not. (user.js is also documentable).

There have been plenty of justifications/use-case arguments here for why to use user.js, and i don't have much more to add, other than i see a lot of single-user-centric thought going on in firefox development these days that ignores how enterprises and workgroups and folks with multiple machines/location who don't want to buy into 'sync' use it.

I want to create useful defaults for users at the enterprise level, but give them the freedom to alter those settings if they aren't done for security policy reasons (lockPref()). Allowing a user to create a user.js is one of the best ways to do this. My users do NOT have access to the APPDIR/INSTDIR (at least on Linux, nor if they don't have ADMIN rights on Windows/MacOS). so how can they modify that? they can't. but they can modify ~/.../user.js. (and no, letting users install their own copies of firefox in their homedirectories is not an answer -- that's another security debacle)

Also, as someone else pointed out Import/Export of config for exchange to prefs.js (or such) is much more likely to cause problems because the entire configuration space is being dumped. Deprecated settings will persist, whereas user.js will presumably have only preferences applicable to the needs of the user.

As for telemetry... If i turn it off/opt-out -- it's likely because i want customization, so you aren't weighing my desires for such customization.

thanks for your consideration.
--stephen

(In reply to sdowdy from comment #32)

$ strace -f -e trace=stat -c firefox -new-instance -profilemanager
% time     seconds  usecs/call     calls    errors syscall                                                                                                                                            
------ ----------- ----------- --------- --------- ----------------                                                                                                                                   
100.00    0.011346           5      2205      1548 stat                                                                                                                                               
------ ----------- ----------- --------- --------- ----------------                                                                                                                                   
100.00    0.011346                  2205      1548 total       

(IGNORING the 40 subprocesses spun off during that interval that i'm sure are consuming a lot more wallclock-time)

Is this just for profile manager and then quitting? Because that's certainly not expected - I'd expect a fraction of that number of stats, and 0 subprocesses, maybe 1 on Windows (which this presumably isn't given strace) for the launcher process. Please file a separate bug so we can investigate. Details about what these stats are would help.

Flags: needinfo?(sdowdy)

I can confirm. They're threads, not necessarily processes. Most of the stat() calls are made for icons in ~/.local/share/icons and /usr/share/icons/gnome.

(In reply to Laurentiu Nicola from comment #34)

I can confirm. They're threads, not necessarily processes. Most of the stat() calls are made for icons in ~/.local/share/icons and /usr/share/icons/gnome.

Please file a separate bug with more details. This seems to be a linux issue. The Firefox profiler's IO marker support is different across platforms (ie not all of stat/open/read/write/close are supported everywhere), so it's possible we don't have profiler stat support on Linux and if no other operations on the file happen, that could explain why we've missed this - I wouldn't know for sure off-hand, but I don't see similar stats listed in https://searchfox.org/mozilla-central/source/browser/base/content/test/performance/browser_startup_mainthreadio.js, and I'd expect them if they happened everywhere. It's also possible that they only happen under certain conditions. Our focus when profiling has been on Windows, as that's where most of our userbase is, but 1000s of stats on icon files seems... unhelpful... during startup. Either way, a bug filed with more details would really help.

Flags: needinfo?(laurentiu)

(In reply to :Gijs (he/him) from comment #35)

Please file a separate bug with more details.

I just filed bug 1579607 (and left it untriaged, as I don't know what it should be under).

Please remove the needinfo? from laurentiu@gmail.com, as that should probably have been me, but it's not me.

(In reply to Laurentiu Nicola from comment #36)

(In reply to :Gijs (he/him) from comment #35)

Please file a separate bug with more details.

I just filed bug 1579607 (and left it untriaged, as I don't know what it should be under).

Thanks, will take a look.

Please remove the needinfo? from laurentiu@gmail.com, as that should probably have been me, but it's not me.

D'oh. Thanks for pointing this out. Going to leave the ni for sdowdy for now, as I suppose there's no guarantee that the stats they're seeing are the same ones you're reporting...

Flags: needinfo?(laurentiu)

(In reply to :Gijs (he/him) from comment #33)

(In reply to sdowdy from comment #32)
...d expect a fraction of that number of stats, and 0 subprocesses, maybe 1 on Windows (which this presumably isn't given strace) for the launcher process. Please file a separate bug so we can investigate. Details about what these stats are would help.

Hmm, i can't reply via e-mail to the needinfo request, it seemed to blackhole (i got two messages, one said do-not-reply, i replied to the other that didn't say that

that's JUST for spinning up profile-manager and exiting [EXIT] (w/o creating a profile)

Here is opening profilemanager and creating a new profile and having it display primary window, then EXIT.

$ strace -f -e trace=stat -o /tmp/strace.log firefox -new-instance -profilemanager
$ wc -l /tmp/strace.log
10968 /tmp/strace.log

Of course a LOT of this is just library (ld.so) lookups, font lookups, various X11/KDE/Gtk resource probes, etc...
(I assume MacOS might have similar issues, but perhaps Windows will have a lot of that stuff pre-determined in the user environment, so hitting disk might not be as often)

My point is those many stat()s *(successful and not) happen regardless, so 1 stat() isn't going to make much difference.

Here's a count of APPDIR references (automounter for /usr/local/firefox-69.0/ ..)

$ grep '/var/autofs/.*firefox' /tmp/strace.log | wc -l
109

$ grep '/var/autofs/.*firefox' /tmp/strace.log | wc -l
109

stat()s on my profile directory:

$ grep '3433jwpa.bugzilla-1543752' /tmp/strace.log | wc -l
3513

Again, ONE stat in this context is noise. Here's some more interesting stat() results:

$ grep -c 'stat(' /tmp/strace.log
10165

As Laurentiu mentions, yes, there's a LOT of icon references, these are going to ultimately be variant based on user environment. same with fonts. for me...

$ grep -c 'stat(.*icons' /tmp/strace.log
3620
$ grep -c 'stat(.*fonts' /tmp/strace.log
1964
$ grep -c 'stat(.*python' /tmp/strace.log
211

eh? didn't realize Python was involved here.

$ grep -c 'stat.*cert9.db' /tmp/strace.log
1757

OUCH

$ grep -c 'stat.*key4.db' /tmp/strace.log
359

$ grep -c '/storage/permanent' /tmp/strace.log
340
$ awk -F'"' '//storage/permanent/{print $2}' /tmp/strace.log | sort | uniq -c | sort -nr | head -6
165 /home/sdowdy/.mozilla/firefox/3433jwpa.bugzilla-1543752/storage/permanent/chrome/idb/3870112724rsegmnoittet-es.sqlite
27 /home/sdowdy/.mozilla/firefox/3433jwpa.bugzilla-1543752/storage/permanent/chrome/idb/3870112724rsegmnoittet-es.sqlite-wal
27 /home/sdowdy/.mozilla/firefox/3433jwpa.bugzilla-1543752/storage/permanent/chrome/idb/3870112724rsegmnoittet-es.sqlite-journal
18 /home/sdowdy/.mozilla/firefox/3433jwpa.bugzilla-1543752/storage/permanent/chrome/idb/3561288849sdhlie.sqlite
18 /home/sdowdy/.mozilla/firefox/3433jwpa.bugzilla-1543752/storage/permanent/chrome/idb/2918063365piupsah.sqlite
18 /home/sdowdy/.mozilla/firefox/3433jwpa.bugzilla-1543752/storage/permanent/chrome/idb/1657114595AmcateirvtiSty.sqlite

Then there's the obligatory repeated localtime/resolv.conf lookups i often see:
$ grep -c '/etc/localtime' /tmp/strace.log
169
$ grep -c '/etc/resolv.conf' /tmp/strace.log
50

I was hoping i could do something like 'firefox -P {profile} -remote 'quit()' (for programmatic testing), but...
$ firefox -new-instance -P bugzilla-1543752 -remote "openURL(www.mozilla.org, new-tab)"

that used to work, but appears to have been removed[1] -- it instead ultimately exits with SEGFAULT:
(sounds like that needs to be fixed)

[1767, Main Thread] ###!!! ABORT: file resource://gre/modules/TelemetryEnvironment.jsm, line 594
[1767, Main Thread] ###!!! ABORT: file resource://gre/modules/TelemetryEnvironment.jsm, line 594
ExceptionHandler::GenerateDump cloned child 1890
ExceptionHandler::SendContinueSignalToChild sent continue signal to child
ExceptionHandler::WaitForContinueSignal waiting for continue signal...
$ echo $?
11

so, that's fun.

--stephen

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1080319

Flags: needinfo?(sdowdy)

(In reply to sdowdy from comment #38)

(In reply to :Gijs (he/him) from comment #33)

(In reply to sdowdy from comment #32)
...d expect a fraction of that number of stats, and 0 subprocesses, maybe 1 on Windows (which this presumably isn't given strace) for the launcher process. Please file a separate bug so we can investigate. Details about what these stats are would help.

Hmm, i can't reply via e-mail to the needinfo request, it seemed to blackhole (i got two messages, one said do-not-reply, i replied to the other that didn't say that

Yeah, email reply on bmo is not a thing, never has been AFAIK.

that's JUST for spinning up profile-manager and exiting [EXIT] (w/o creating a profile)

Thanks for clarifying this, and the follow-up investigation. I will be poking at some of these and filing follow-up bugs where appropriate. Just to briefly respond to some of your points:

My point is those many stat()s *(successful and not) happen regardless, so 1 stat() isn't going to make much difference.

At least on Windows, this is not the case - we see such stats taking literally >100ms on slower (non-SSD) machines. In terms of Firefox startup time, that's a lot of time that could be spent more productively. Similarly, we care a lot more about startup IO than about IO that happens "later". That is, ideally Firefox should do the absolute minimum of IO prior to showing you a window and loading the homepage / restored primary tab's page.

AFAICT, the strace metrics also do not account for which threads do the IO. Doing background reads is a lot less problematic - I'd expect the sqlite ones to fall under those, for instance (ie I expect them to not block the main Firefox process starting up), and in fact they are likely to be happening in the child process, rather than the main process, too. Of course, there could be disk contention, but so far this seems much less of a problem than the main thread simply being stuck waiting for IO (certainly on slower machines).

There's already a list in https://searchfox.org/mozilla-central/source/browser/base/content/test/performance/browser_startup_mainthreadio.js of IO that we know happens on the main thread when Firefox starts, including data about which platforms it happens on and during which phase of startup it happens. As you can see from the test, we have bugs on file for pretty much everything that isn't inevitable, so we're not singling out user.js unduly - it's just that, like all the other IO that we filed bugs for, it serves no point for the majority of users. Note that this is the nightly copy of the test, so some items will have been removed that you'd still see in strace logs of release.

The additional Linux data is very helpful, though it will need some refining before it can be used. Really, we also need to add stat support to the linux file hooks so that we can get this data into that automated test (so we find out if more IO is added).

<snip>
it instead ultimately exits with SEGFAULT: (sounds like that needs to be fixed)

[1767, Main Thread] ###!!! ABORT: file resource://gre/modules/TelemetryEnvironment.jsm, line 594
[1767, Main Thread] ###!!! ABORT: file resource://gre/modules/TelemetryEnvironment.jsm, line 594

This is shutdown timing out while waiting for the add-on manager. It looks like it's tracked in bug 1513855.

I started digging through the strace stuff and filed a metabug at bug 1580658 to keep track. If you have further suggestions for where to take that, please comment there, so we can keep this bug on-topic, inasmuch as that's possible.

You need to log in before you can comment on or make changes to this bug.