1951642 - Decide on a path forward for preprocessed resource: modules wrt moz-src

Status: NEW

(Firefox :: General, task, P3)

Description

[:Gijs (he/him)]

There are a few modules (specifically, CrashManager.sys.mjs, RFPTargetConstants.sys.mjs, PromiseWorker.mjs, PromiseWorker.js) that use objdir paths in moz.build combined with EXTRA_JS_MODULES. There are a handful of other uses of EXTRA_PP_JS_MODULES which also use preprocessing, which will also include gems such as AppConstants.sys.mjs.

We haven't finalized what we want to do with these. Although from a packaging perspective, it would be straightforward to make these accessible using the moz-src protocol as-is (all that needs to happen is that the processed file ends up in the right location), the bigger question IMO is how we'd expect this to work with source code resolvers etc., and based on that, if we want some kind of signifier (either in the host portion of the URL, the filename, or the protocol itself) that the file is preprocessed.

It's likely that some of our automated tools (likely eslint, prettier and any typescript support) cannot process the original input files or scripts (with #ifdef or other generation magic in them - they might just be python files!). However, for other tools, notably in-editor and in-searchfox link support and similar, being able to effectively map these URLs to their input files would be useful. So I expect we want some way of distinguishing these files.

Originally we thought this would use a separate protocol or separate host. The current plan is that we use the host portion of the URL instead for things like "these files should only be accessible to about: pages" and other security and/or process restrictions. Also using it for whether the file is generated or not may be possible but feels like it runs into issues if we want to use generated files exposed only to about: pages or in specific processes. So my gut is that we would want to encode any info we want to in some other way - either the filename or the protocol. Adding protocol support for moz-src was a bit tedious (lots of if conditions that needed to account for chrome, resource and now also moz-src), and adding another protocol to that also feels like it exacerbates the "here's another odd thing you have to learn about our codebase" problem.

So I wonder about instead using filenames for this. We already do something similar for worker files, where we use .worker.js/.worker.mjs to allow us to easily give eslint the right concept of the environment in which those scripts run. The main downside I see is that this will likely mean exported symbols no longer match the filename, and that it will bulk out import statements even for cases where the consumer doesn't care the file is generated. Of course there is no requirement that symbols match the filenames, the whole thing just feels a bit yucky.

Do other folks have different ideas on how to approach this, and/or does this sound reasonable?

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Toolkit::UI Widgets' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Nika Layzell [:nika] (ni? for response)]

I was originally thinking that objdir files would not be on a separate protocol, but rather than the objdir files would effectively "overlay" on top of the source directory, meaning that the generated file would be used as priority over the source tree file when doing a lookup (as it would be the one included in the bundle). This seems to me like it would be something other organizations would also want (to allow things like pre-processed files existing alongside normal source directory files). Is this sort of thing not supported by the tools we use?

My main worry with having a separate protocol is the amount of code churn which would be required as a file is moved to either need some form of pre-processing or moved away from needing pre-processing as time goes on. If say XPCOMUtils.sys.mjs or Logging.sys.mjs were changed between these two states it could require a lot of files to be updated.

Comment 3

[Tomislav Jovanovic :zombie]

However, for other tools, notably in-editor and in-searchfox link support and similar, being able to effectively map these URLs to their input files would be useful.

We already had to come up with an approach for things like AppConstants.sys.mjs. If you open Extension.sys.mjs line 247 in VSCode and control-click on the .platform field, it will jump to the definition in a (manually written) substitution file.

From your comment, it sounds like it could be useful to have a link from that file to the actual source file behind it. That sounds like a good idea in general (and I would have probably thought of adding it if only relative source links worked in VSCode on #@$'in Windows).

I expect we'll end up with a mapping in jsconfig.json for each of the preprocessed files (ignore the fact it's currently in index.d.ts, that's temporary until the moz-src thing is settled), and that could be used by all tools which want to parse the code (typescript, editors, searchfox).

So I don't think I have an insightful opinion on protocol vs name suffix, either could work as long as we a way to do the mapping.

Comment 4

[Dave Townsend [:mossop]]

I don't have a strong opinion and would expect the number of such files to be small enough that it is likely fine to do manual work to solve this. That could be writing mappings into jsconfig.json if all tooling supports that, it could also be writing a stub file with the same path in the source directory as the generated file claims which then exports enough to make it work for tooling plus a comment at the top explaining where how the file is actually generated.

Comment 5

[Mark Banner (:standard8)]

I don't have anything extra to add.